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This  report  provides  and  documents  the  results  of  an  investigation  into  the 
policies  and  procedures  requirements  for  automatic  test  equipment.  The  candidate 
areas  for  investigation  were  selected  on  the  basis  of  their  potential  as  a "cost  driver" 
in  ATE  programs  or  items  which  have  been  identified  in  the  Report  on  Navy  Issues 
Concerning  Automatic  Test,  Monitoring  and  Diagnostic  Systems  and  Equipment. 


Six  basic  issues  or  problem  areas,  addressed  in  the  summary,  were  reviewed 
during  the  investigation.  This  policies  and  procedures  analysis  and  the  conclusions  and 
recommendations  provided  are  the  result  of  an  extensive  review  of  standards  (existing 
and  proposed),  instructions  and  ATE  related  specifications.  Data  and  recommendations 
concerning  the  data  base  documents  were  considered  too  lengthy  to  be  included  in  the 
basic  text.  This  information  is,  however,  considered  to  be  valuable  and  pertinent  to 
the  effort  and  as  such  has  been  included  as  appendicies  A through  H.  A brief  summary 
of  each  app>endix  is  provided  as  follows: 

Appendix  A - NAVMAT  ATE  Instructions 

This  appendix  deals  with  ATE  instructions  generated  by  NAVMAT.  These 
instructions  have  been  categorized  into  nine  separate  areas,  each  having  an  ATE  area 
of  impact.  A summary  of  each  instruction  is  provided  with  recommendations  for 
further  improvement  as  applicable.  A cross  correlation  with  other  instructions  and 
reference  documentation  is  provided. 

Appendix  B - DoD  Test  and  Monitoring  Systems  Policies  and  Procedures 

The  ATE  policies  and  procedures  activities  of  DoD,  Air  Force,  Army,  OPNAV, 
NAVAIR  and  NAVELEX  were  included  in  this  review.  This  appendix  provides  a 
composite  listing  of  ATE  related  documents  by  responsible  organization,  with  each 
document  defined  in  a broad  general  category  (i.e.  logistics,  software  and  configur- 
ation management,  etc.). 

Appendix  C - ATE  Related  Military  Specifications 

On-line  testing  considerations  as  an  extention  of  ATE  which  are  covered  by 
military  specifications  were  reviewed.  In  particular,  the  specifications  which 
addressed  BIT,  interface  bus  and  test  point  specifications  were  highlighted.  A top- 
down  specification  hierarchy  is  provided  in  addition  to  the  interrelationship  of  these 
specifications  and  their  impact  on  the  key  discussion  areas.  Comments  and 
recommendations  in  each  area  are  provided,  as  well  as  a description  of  the  major 

siibjert  area  covered  bv  the  document.  , 
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Appendix  D - Revised  NMC  Program  Plan  Event  Chart 


This  brief  pictorial  appendix  relates  to  the  six  major  ATE  task  areas  covered  in 
this  policies  and  procedures  analysis.  The  policy  and  procedure  documents  affecting 
each  area  have  been  reviewed,  with  recommendations  annotated. 

Appendix  E - Excerpts  From  Report  of  Industry  Ad  Hoc  Automatic  Test 

Equipment  Project 

Industry  recommendations  for  Operational  Readiness  Monitoring  Systems  (ORMS) 
and  utilization  of  on-line  and  test  point  implementation  for  ORMS  are  provided.  Also 
identified  are  the  needs  for  standards  for  ORMS  test  points  and  a proposed 
implementation  plan. 

Appendix  F - Recommended  NAVMAT  Instructions 

A review  of  ATE,  TPS  acquisition  portions  of  B-1  and  S3  A Programs  were 
reviewed.  This  review  revealed  the  need  to  acquire  the  analytical  data,  tools  and 
procedures  utilized  in  making  trade-off  and  tester  selection  studies  and  those  which 
establish  management  controls  on  those  programs.  Samples  of  DIDs  and  CDRLs  are 
provided  as  a springboard  for  obtaining  the  data  necessary  to  accomodate  a managerial 
overview  into  the  rational  utilized  in  forming  the  final  acquisition  decisions. 

Appendices  G,  H,  I,  - Policies  and  Procedures  Analysis  for  Conceptual,  Validation 

and  Full  Scale  Development  Phases 

These  three  appendicies  deal  with  ATE  acquisition  related  activities.  These 
activities  are  based  on  the  Acquisition  Guide  for  Automatic  Test,  Monitoring  and 
Diagnostic  Systems  and  Equipment,  and  are  in  a work  breakdown  structure  format  for 
each  acquisition  phase.  Each  phase  provides  a detailed  ATE  procurement  process,  time 
phased  to  the  acquisition  processes  and  keyed  to  the  weapon  system  procurement 
milestones.  Each  appendix  also  provides  short  term  and  long  term  recommendations 
for  each  procurement  phase. 

An  Annex,  which  supplements  the  Full  Scale  Development  Work  Breakdown 
Structure,  is  provided  to  deal  solely  with  the  ATE  selection  process,  data 
requirements,  compatibility  analysis,  and  cost  predictions. 

Attachment  1 provides  a brief  summary  of  Test  Requirement  Document  (TRD) 
objectives  necessary  to  assemble  a comprehensive  data  package  which  will  form  the 
basis  of  and  provide  the  baseline  data  for  automatic  test  equipment  decisions. 

The  analysis  undertaken  centered  on  a systems  engineering  "macro"  or  overview 
approach.  The  problem  areas  were  addressed  in  terms  of  the  real  world  requirements 
for  automatic  test  systems. 


2 


I 


< 


The  actions  necessary  to  improve  the  performance  of  the  total  acquisition 
process  are  identified  as  specific  policy  and  procedure  requirements  necessary  to 
assure  efficient  selection  and  utilization  of  automatic  test  systems.  The  following 
provides  a brief  summary  of  the  recommendations  for  each  area  reviewed: 

Test  Program  Set  Acquisition 

o Develop  a TPS  Handbook  which  addresses  TPS  development  as  an 
integrated  process  and  provides  guidance  and  direction  concerning  CAD  and 
simulator  applications.  Quality  control  for  TPSs  to  assure  a higher  degree 
of  confidence  in  the  TPS's  ability  to  satisfy  the  fault  isolation/support 
requirements. 

o Establish  a work  package  concept  for  TPS  development.  This  concept 
should  encompass  staffing  on  a team  basis  from  initial  analysis  through 
final  sell  off  and  schedule  integrated  with  the  deployment  forecasts. 

o Develop  a revised  AR-9  which  will  serve  as  a Navy  wide  (vice  NAVAIR) 
TPS  guide.  This  document  should  address  the  TPS  development  process 
from  a time  phased  standpoint  to  assure  a cohesive  program. 

o Establish  a policy  which  will  require  application  of  computer  aided  design 
(D-LASAR)  and  simulation  models  for  TPS  development.  This  policy  must 
address  the  latest  available  technology  (e.g.,  LSI,  microprocessors). 

o Establish  a policy  which  requires  that  TPS  developments  be  funded  from 
the  prime  contract  and  that  TPS  acquisition  plans  be  integrated  as  key 
milestones  into  the  prime  system  contract.  Economies  in  the  prime  system 
contract  should  not  be  made  in  the  TPS  procurement  area. 

o Establish  a TPS  management  guideline  position  which  addresses  the  TPS 
development  process  in  terms  of  staffing  work  load,  validation,  verification 
and  work  breakdown. 

o Establish  a policy  requiring  that  TPS  developments  be  awarded  on  a 
competitive  procurement  basis. 

Automatic  Test  Equipment  Planning 

o Provide  immediate  guidance  for  ATE  acquisition  managers  based  on  the 
methodology  proposed  in  the  Acquisition  Planning  Guide  for  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment  shown  as  the 
Annex  of  Appendix  1. 

Require  compulsory  utilization  of  ATE  data  banks  as  an  input  to  the  ATE 
selection  process.  This  requires  that  the  data  bank  I/O  specifications, 
access  times,  currency  of  information  and  cost  data  be  validated  prior  to 
directing  this  utilization. 


o Establish  methods  for  producing  computerized  and  non-computerized 

support  analysis.  Computerized  analysis  such  as  the  TEEM  model  will 
require  updating  to  address  the  deficiencies  as  shown  in  Appendix  G. 

o The  non-computerized  analysis,  provided  by  MIL-STD-1513,  requires 

updating  to  account  for  step  function  work  load  analysis,  queueing  impacts 
as  a function  of  step  input  inrush,  and  optimization  rationale  for  ATE 
selection. 

o Establish  that  BIT  is  a viaoie  support  methodology  and  that  the  impact  of 
BIT  at  each  support  level  and  in  support  equipment  selection  (ATE  & MTE) 
must  be  considered  and  reflected  in  the  LCC. 

o Establish  that  a review  of  MIL-STD-1326  in  conjunction  with  other  BIT/test 
point/interface  bus  related  documents  is  required  in  the  ships  community. 
This  review  should  result  in  a new  testability  standard.  This  standardized 
standard  should  incorporate  subsystem  design  criteria  for  utilizing  test 
points  for  ORMS  and  assure  that  ORMS  and  SDMS  compatibility  is 
established. 

Test/Repair  Documentation 

o Establish  that  the  level  of  support  documentation  necessary  for  adequate 
support  be  established  in  conjunction  with  adoption  of  the  support  concept. 

o 'Require  that  support  documentation  be  developed  in  concert  with  the 
systems  development,  and  that  acceptance  of  the  systems  hardware  be 
predicated  on  delivery  of  an  acceptable  documentation  package. 

o Review  and  upgrade  the  documentation  specifications  (e.g.  MIL-D-1000)  to 
assure  that  documentation  is  provided  in  a uniform,  consistent  manner. 

Integrated  Logistics  Support 

o Develop  an  improved  TEEM  model  (as  reflected  in  Appendix  G)  to 
accommodate  a real  world  support  situation,  provide  BIT  related  cost 
factors,  all  user  interaction  for  alteration  of  assumed  values  and  queueing 
analysis  for  more  than  one  UUT  across  a given  tester. 

o Establish  an  ILS  checklist  which  allows  project/program  managers  to 
benchmark  the  support  area  in  a time  phased  manner. 

o Establish  that  economies  in  the  prime  equipment  are  not  to  be  made  at  the 
expense  of  the  support  equipment.  This  can  be  accomplished  by 
establishing  a companion  ILS  for  support  equipment  which  has  funding 
derived  from  but  independent  of  the  prime  system  procurement. 

o Establish  that  the  prime  and  support  system  ILS  (i.e.,  companion)  be 
integrated,  and  mutually  justified  prior  to  DSARC. 
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Contractual  Requirements 

' o Establish  a policy  which  institutes  the  requirement  to  have  ATE  selection 

I trade-off  analysis,  and  the  effect  of  BIT/ATE  on  systems  maintenance 

repair/throwaway  justifications  supplied  as  deliverables.  This  will  provide 
the  proper  incentive  to  contractors  to  perform  adequate  analysis 
concerning  maintenance  posture  and  support  equipment  selection. 

Configuration  Control 

V , o It  is  necessary  to  establish  a configuration  control  document  which 

■ addresses  the  test  program  set  as  an  entirety  (i.e.,  test  system,  ID,  UUT, 

documentation  and  programs).  To  this  end,  it  is  recommended  that  an 
upgrading  and  integration  of  MIL-STD-480,  481,  NAVAIR  Instruction  4120.1 
Appendix  C be  accomplished  to  ensure  a consistent  configuration  control 
^ posture. 
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1.0  BACKGROUND 

There  has  been  a significant,  continuing  effort  within  the  Department  of  Defense 
and  the  Navy  to  standardize  the  selection,  acquisition,  and  implementation  procedures 
for  automatic  test  equipment  (ATE).  A great  many  directives,  policies,  procedures  and 
other  documents  reflect  a strong  determination  to  reduce  the  proliferation  of  support 
equipment  and  the  associated  high  costs  of  acquiring  and  maintaining  this  equipment. 
An  integration  of  the  ATE  design,  development,  selection,  and  acquisition  processes 
with  the  integrated  logistics  support  (ILS)  and  weapon  systems  acquisition  processes  is 
mandatory  if  this  goal  is  to  be  achieved. 

As  a result  of  the  above  and  the  issues  raised  in  the  Report  on  Navy  Issues 
Concerning  Automatic  Test  Monitoring  and  Diagnostic  Systems  and  Equipment,  the 
Marcy  Report,  it  became  apparent  that  a thorough  review  of  Navy  policies  and 
procedures  for  the  acquisition  and  utilization  of  ATE  within  the  Navy  was  required. 

A review  of  NAVMAT,  NAVAIR,  NAVELEX,  NAVSEA,  SECNAV,  OPNAV, 
NAVSHIPS,  Army,  Air  Force  and  DoD  policies  and  procedures  relating  to  test  and 
monitoring  systems  (TAMS)  was  undertaken  to  determine  areas  of  deficiency  in  the 
acquisition  of  TAMS.  Also  to  be  reviewed  were  TAMS  related  military  specifications. 
Appendices  A through  C contain  details  of  this  review  and  include  policies  and 
procedures  descriptions. 

The  result  of  this  review  was  Table  1 which  lists  TAMS  and  related  applicable 
documents,  those  which  need  revisions,  what  the  revisions  should  cover  (Appendix  A) 
and  what  new  policies  and  procedures  are  required. 

The  review  also  covered  all  existing  DoD  instructions  and  MIL-SPECs  relating  to 
ATE.  The  items  reviewed  are  provided  in  Appendix  B,  with  applicable  military 
specifications  and  their  interrelationships  provided  in  Appendix  C.  This  review 
resulted  in  the  formulation  of  the  requirement  for  and  the  approach  to  the  policy  and 
procedure  areas  addressed  herein. 

2.0  INTRODUCTION 

The  purpose  of  the  policies  and  procedures  task  item  is  to  provide  recommenda- 
tions for  review,  revision,  and  generation  of  new  Navy  policies  and  procedures/tools 
(e.g.,  MIL-STDs,  design  to  guides,  etc.)  which  will  furnish  guidelines  and  assistance  to 
both  Navy  and  industrial  project  managers  in  the  off-line  ATE  and  built-in-test  (BIT) 
design,  development,  selection,  and  acquisition  processes.  The  policies  and  procedures 
and  tools  developed  in  response  to  these  recommendations  will  provide  standardized 
management  tools  which  will  assist  Navy  project  personnel  and  industrial  contractors 
in  delivering  testable  systems  at  an  affordable  cost. 
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TABLE  1.  TAMS  AND  OTHER  RELATED  DOCUMENTS 


THE  NAVMAT  DOCUMENTS  LISTED  ARE  DIRECTLY  APPLICABLE  TO  TAMS.  WITH  REVISION 
REQUIREMENTS  AS  NOTED. 


INSTRUCTION 

NUMBER 

DESCRIPTION 

REVISION 

3960.4A 

POLICY  AND  RESPONSIBILITY  FOR 
AUTOMATIC  TEST  MONITORING  AND 
DIAGNOSTIC  SYSTEMS,  (ATMDS), 
IMPLEMENTATION  OF 

NONE 

3960.9 

ACQUISITION  PLANNING  GUIDE  FOR 

ATMDS  AND  EQUIPMENT:  BIT  DESIGN 
GUIDE,  PROMULGATION  OF.  (BOTH 
DOCUMENTS  REQUIRE  PERIODIC  UP- 
DATING.) 

THE  FOLLOWING  SHOULD  BE 
INTEGRATED  INTO  THE  GUIDE; 

• ATE  SELECTION 

• ATE  WORKLOAD  ANALYSIS 

• TEST  REQUIREMENTS 

ANALYSIS 

• TESTABILITY  REQUIREMENTS 

4440.46 

INVENTORY  OF  ATMDS,  REQUEST  FQR 

NONE 

4120.10SA 

ATE  PROGRAMMING  LANGUAGE, 

STANDARD  FOR 

NONE 

5230.8 

DATA  BANKS  FOR  ATMDS  8i  EQUIP- 
MENT. UTILIZATION  OF 

NEEDS  UPDATING.  COMBINE 

DATA  BANK  AND  INVENTORY 
INSTRUCTION.  COVER-CONTROL, 
FOCAL  POINT,  MIL-STD-1388, 

ATE  CHARACTERISTICS.  CON- 
TENT-LOGISTICS MODELS,  INVEN- 
TORY R&D  REPORTS,  GIDEP, 
1634-DDC,  NOSC,  HOW  TO 

UPDATE,  ETC. 

4790.16 

DOD  EOUIPMENT  MAINTENANCE  PROGRAM 

REFERENCE  ATE  DATA  BANKS 

P-4000 

ILS  IMPLEMENTATION  GUIDE  FOR  DOD 
SYSTEMS  AND  EQUIPMENTS 

REVIEW  FOR  ATE  COVERAGE 

4000.29 

BASIC  PRINCIPLES  OF  LOGISTICS 

REVIEW  FOR  ATE  COVERAGE 

5420.44 

TMDE  SUPPORT  ADVISORY  COMMITTEE; 
ESTABLISHMENT  OF 

REWRITE  ATE  SECTION  TO 
INCLUDE  PERIODIC  MEETINGS 

BY  ATE  FOCAL  POINTS 

7000.1 9A 

COST  ANALYSIS  PROGRAM 

INCLUDE  SPECIAL  REQUIRED 

ATE  SOFTWARE  COST  FORMATS 
AND  REPORTING  GUIDES 
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INSTRUCTION 

NUMBER 

DESCRIPTION 

REVISION 

AR8  (NAVAIR) 

V/VST/AVIONICS  SYSTEMS  COMPATIBILITY, 
GENERAL  REQUIREMENTS  FOR 

BEING  REWRITTEN  AS  MIL- 
STANDARDS  - COORDINATE 

NEEDS  TO  BE  NAVYWIOE 

AR9/AR9-B 

(NAVAIR) 

VAST  TEST  PROGRAMS,  GENERAL 
REQUIREMENTS  FOR 

BEING  REWRITTEN  AS  MIL- 
STANDARDS  - COORDINATED 
NEEDS  TO  NAVYWIDE.  ALSO 
REQUIRES  BETTER  TPS  VALI- 
DATION AND  STATISTICAL 

TOOLS. 

ARID 

(NAVAIR) 

MAINTAINABILITY  OF  AVIONICS 

EQUIPMENT  AND  SYSTEMS,  GENERAL 
REQUIREMENTS  FOR 

BEING  REWRITTEN  MIL- 
STANDARD  - COORDINATED 
NEEDS  TO  NAVYWIDE. 

MIL-STD-1309 

DEFINITIONS  OF  TERMS  FOR  TMDE 

NEEDS  REWRITE  TO  EXPAND 

AND  CLARIFY 

MIL-STD-1326 

TEST  POINTS,  TEST  POINT  SELECTION 

AND  INTERFACE  REQUIREMENTS  FOR 
EQUIPMENT  MONITORED  BY  SHIPBOARD 
ON-LINE  ATE 

RECOMMEND  REVISION.  TO 
INCLUDE  DATA  BUS,  BIT  REQTS. 

MIL-STD-1399 

SHIPBORNE  SYSTEM/EOUfPMENT 
ACQUISITION  MANUAL 

APPENDIX  31  (SUPPORT  AND 

TEST  EQUIPMENT)  REQUIRES 
UPDATING  TO  INCLUDE  ORMS/ 

BIT,  ETC. 

NAVMAT  INST 
4120.10/A 

IMPLEMENTATION  OF  MIL-STD-1399 

INCLUDE  MORE  GUIDANCE  ON 

ATE  SELECTION  PROCESS 

MIL-STD-1513 

(USAF) 

TRADE  STUDIES  FOR  THE  SELECTION 

OF  AVIONIC  SUPPORT  SYSTEMS, 

CRITERIA  FOR 

NAVMAT  INST 
3960.9 

BIT  DESIGN  GUIDE 

EXPAND  AND  PROVIDE  GUIDE 

ON  BIT  VERIFICATION,  VALIDA- 
TION PROCEDURES  & STATIS- 
TICAL TOOLS 

TEST  PROGRAM  DESIGN  GUIDE 

EXPAND  AND  PROVIDE  MORE 
GUIDANCE  ON  TPS  VERIFICA- 
TION & VALIDATION  PRO- 
CEDURES. 

PROVIDE  BETTER  TPS  VERIFI- 
CATION AND  VALIDATION 
STATISTICAL  TOOLS. 
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3.0  APPROACH 

A systems  engineering  approach  to  this  task  item  was  adopted.  The  approach 
taken  was  to  refine  and  augment  in  a systems  engineering  fashion  the  functional  flow 
of  ATE  related  acquisition  and  development  activities  as  depicted  in  Figure  1-1  of  the 
Acquisition  Planning  Guide  for  Automatic  Test,  Monitoring  and  Diagnostic  Systems  and 
Equipment.  The  material  contained  in  this  guide  is  structured  to  assist  program 
managers,  ATE  acquisition  managers  and  others  associated  with  the  acquisition  of 
Navy  hardware,  both  in  government  and  industry,  in  relating  ATE  acquisition  planning 
requirements  to  activities  undertaken  during  each  phase  of  the  system  acquisition 
process.  For  this  reason,  this  document  was  chosen  as  the  primary  vehicle  from  which 
to  conduct  and  initiate  the  policies  and  procedures  effort. 

The  basic  premise  behind  this  approach  is  that  one  can  not  realistically  propose 
and  write  policy  and  procedures  unless  one  can  describe  the  sub-processes  and  the 
logical  time  phased  relationships  of  the  sub-processes  which  make  up  a system  process. 

The  system  processes  under  discussion  are  the  ATE/BIT  design  and  development 
processes.  Process  flow  charts  for  the  conceptual,  validation,  and  full-scale  1 

development  phases  were  developed  and  can  be  found  in  Appendices  G,  H,  and  1 of  this  i 

report.  i 

The  initial  intent  of  this  effort  was  to  overlay  on  the  sub-processes  of  each  phase 
of  system  acquisition  existing  Navy  policies,  procedures,  instructions  and  tools,  (i.e.,  i 

MIL-STDs,  design  to  guidelines,  GFE  software  aids,  etc.)  and  to  identify  voids  (i.e.,  i 

situations  where  no  tools  existed  to  do  a process).  Then  tri-service  and  industry 
expertise  was  to  be  solicited  and  researched  to  uncover  and  identify  tools  currently  ? 

available  which  will  fill  these  voids.  If  tools  for  sub-processes  could  not  be  identified 
or  if  the  existing  tools  could  not  be  identified,  the  need  for  new  process  tools  would  be 
identified.  Figure  1 provides  an  overview  of  the  approach  taken  for  this  project. 

It  became  obvious  to  the  policy  and  procedures  investigators  that  the 
fragmentation,  overlap,  and  the  general  lack  of  systems  thinking  resident  in  existing 
policy,  procedures,  and  instructions  made  this  task  virtually  impossible  to  undertake. 

That  is,  a clear  "one  for  one"  correspondence  of  existing  instructions  with  ATE 
acquisition  processes  was,  for  the  most  part,  virtually  impossible  to  establish. 

1 

This  situation  dictated  to  the  investigators  that  a "macro"  rather  than  "micro" 
systems  approach  to  policy/procedure  analysis  would  be  more  effective.  The 
investigators,  utilizing  the  augmented  process  flow  charts,  proceeded  to  identify 
policy/ procedure  issues  which  had  major  cost  drive  implications  in  ATE/support  related 
matters.  This  approach,  it  was  felt,  would  bear  more  fruitful  results  than  the  classic 
"overlay"  approach  originally  conceived.  Utilizing  this  approach,  major  policy  and 
procedure  issues  were  then  identified  and  verified  via  liaison  with  select  project  and 
field  offices.  The  reference  to  and  traceability  of  issues  to  the  Report  on  Navy  Issues 
Concerning  Automatic  Test,  Monitoring  and  Diagnostic  Systems  and  Equipment  was 
also  a criteria,  besides  cost,  by  which  major  issues  were  selected. 
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Figure  1 . Policies/Procedures  Objectives. 


4.0  SUMMARY  ANALYSIS 


As  a result  of  applying  the  selection  criteria  defined  in  Section  3.0,  the  following 
major  policy  and  procedure  issues  were  selected  for  analysis  in  this  report: 

o TPS  Acquisition 

o ATE  Planning 

o Test/Repair  Documentation 

o ILS/ATE  Considerations 

o ATE/Contractual  Requirements 

o Configuration  Control. 

A brief  synopsis  of  the  findings  arrived  at  as  a result  of  the  studies  undertaken 
under  the  policies  and  procedures  task  item  follow. 


5.0  ATE/SUPPORT  COST  DRIVERS 

5.1  TPS  acquisition. 

5.1.1  Statement  of  the  problem.  The  complex  process  of  test  program  set  (TPS) 
acquisition  has  been  the  source  of  substantial  criticism  based  principally  on  the 
outcome  of  the  process:  a product  generally  characterized  by  unsatisfactory 
performance  and  acquired  at  unexpectedly  high  costs.  The  unsatisfactory  performance 
characteristics  identified  with  Fleet  use  of  TPSs  have  been  described  in  detail  in  the 
Report  on  Navy  Issues  Concerning  Automatic  Test,  Measurement  and  Diagnostic 
Systems  and  Equipment.  Some  rationale  for  unexpectedly  high  costs  is  provided  in  the 
VAST/TPS  Cost  Investigation  (Provided  to  MAT  04T  under  contract  N00014-76-C- 
1090): 

[ 5.1.2  Analysis.  Both  poor  performance  and  high  cost  are  symptomatic  of  problems  for 

which  solutions  must  be  sought  in  three  distinct  areas  of  activity:  technical, 
management  and  procurement. 

5. 1.2.1  Technical  area.  This  activity  includes  the  definition  of  standard  input 
requirements  to  the  process,  optimum  procedures  and  techniques  for  design  and 
development,  valid  quality  control  criteria  and  practices,  and  standard  output  and 
documentation  requirements  for  TPS  use  and  maintenance. 

a.  Standard  input  requirements.  Standard  test  requirements  have  been 
defined  in  the  test  requirement  documentation  (TRD)  specification  presently  being 
coordinated  by  the  SYSCOMs  for  release  as  a Navy  standard.  This  document  makes 
distinction  between  test  requirements  data  needed  to  support  different  end  use 
' objectives.  Detailed  procedural  data  are  specified  to  satisfy  manual  test  design 

' requirements  for  fault  detection  and  isolation.  Non-procedural  data  are  specified  for 

[ input  to  design  of  screening  tests  and  test  programs  to  be  produced  with  computer- 

I aided  design  (CAD)  tools,  i.e.  D-LASAR.  This  distinction  is  made  because  of  the 

substantial  savings  in  test  requirement  documentation  costs  which  are  possible  by 
specifying  non-procedural  data  when  appropriate.  Policy  guidelines  are  needed  to 
provide  the  general  criteria  for  making  the  non-procedural  decision.  Included  with 
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these  criteria  should  be  a compendium  of  available,  Navy-approved  CAD  systems,  their 
general  capabilities,  and  limits  of  application. 

b.  Design  and  development  techniques.  Although  much  material  is  available 
in  the  automatic  testing  literature  with  respect  to  techniques  and  procedures  which 
can  be  used  to  optimize  the  test  design  and  development  process,  most  of  it  deals  with 
isolated  segments  rather  than  the  process  as  a whole.  One  document  which  treats  the 
process  as  a continuum  is  the  Program  Design  Handbook  for  Automatic  Test  Equipment 
prepared  for  the  Navy  by  RCA  in  1968.  This  document  has  seen  wide  application  and  is 
still  useful.  However,  as  is  apparent  from  the  publication  date,  it  does  not  present 
refined  techniques  and  procedures  which  may  have  developed  in  the  interim.  Of 
specific  concern  are  testing  techniques  which  can  be  translated  into  test  programs  for 
units  under  test  (UUTs)  incorporating  LSI  and  microprocessor-type  devices.  Another 
area  not  adequately  covered  is  the  application  of  CAD  systems  previously  mentioned. 
The  need  for  revision  of  the  document  is  recognized  but  a firm  commitment  to  the 
revision  task  has  not  been  made. 

c.  Quality  control  criteria  and  practices.  As  was  stated  previously,  the 
quality  of  test  programs  arriving  in  the  Fleet  has  been  unsatisfactory.  Quality  control 
for  test  programs  is  exercised  through  the  validation  and  acceptance  functions  based 
on  statistical  sampling  techniques  that  do  not  necessarily  address  the  total  population 
of  faults.  For  example,  acceptance  testing  per  AR-9B  is  based  on  a random  sampling 
of  faults  from  a fault  sample  selection  list.  Although  the  format  for  this  list  is 
prescribed,  the  boundaries  of  it  are  not.  In  short,  the  relationship  of  the  list  to  the 
total  fault  population  is  not  defined.  While  it  may  be  reasonable  to  assume  that  the 
list  would  include  all  faults  for  which  diagnostic  tests  have  been  developed,  this  still 
does  not  provide  a measure  of  comprehensiveness  without  reference  to  some  total 
number  of  possible  faults.  What  results  is  a false  level  of  confidence  that  the  test 
program  will  perform  its  necessary  diagnostic  functions  in  the  specified  support 
environment.  Notwithstanding  this  uncertainty  with  respect  to  the  technical  validity 
of  present  quality  control  procedures,  it  should  also  be  stated  that  because  of  the 
effects  of  operational  requirements  on  TPS  deliveries,  it  is  known  that  these 
procedures  were  not  followed  consistently.  Thus,  there  is  no  certain  correlation 
between  the  poor  TPS  performance  noted  by  the  Fleet  and  the  quality  control 
procedures  as  they  are  currently  stated.  The  obvious  first  step,  therefore,  is  to  ensure 
that  validation  and  acceptance  requirements  are  met  as  specified  in  existing  guidance 
documents,  i.e.,  the  Program  Design  Handbook  for  Automatic  Test  Equipment  and  AR- 
9B. 

d.  Rate  tooling  validation.  An  additional  quality  control  effect  can  be 
realized  through  the  implementation  of  the  rate  tooling  concept.  This  concept 
employs  the  same  ATE/TPSs  developed  for  maintenance  in  place  of  factory  test 
equipment  to  perform  incoming  inspection/acceptance  of  avionics  at  the  integrating 
contractor's  facility.  This  increases  the  exposure  given  to  test  programs  beyond  that 
attained  during  validation  and  acceptance  and  thus  provides  an  opportunity  to 
eliminate  latent  problems  to  the  greatest  extent  possible  before  the  TPS  reaches  the 
Fleet. 
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e.  TPS  output  and  documentation  requirements.  Standard  output  and 
documentation  requirements  for  TPS  use  and  maintenance  are  specified  in  AR-9B.  In 
general,  these  requirements  relate  primarily  to  the  content,  organization  and  delivery 
of  TPS  elements  used  with  VAST.  AR-9B  should  be  revised  to  accommodate  standard 
TPS  requirements  in  broader  applications. 

5. 1.2. 2 Management  area.  This  activity  includes  requirements  associated  with 

planning,  organizing  and  controlling  the  TPS  acquisition  process. 

a.  TPS  acquisition  planning.  No  specific  requirements  for  planning  the  TPS 
acquisition  process  are  identified  in  any  existing  guidance  or  policy  documents, 
although  the  requirement  is  inferred  as  part  of  the  off-line  ATE  support  plan  in  the 
Acquisition  Planning  Guide  for  Automatic  Test,  Monitoring  and  Diagnostic  Systems  and 
Equipment.  The  need  for  projection  of  key  events  in  a large-scale,  complex 
undertaking  such  as  TPS  acquisition  is  obvious;  the  time  phased  relationships  between 
the  availability  of  TRDs,  UUT  hardware,  trained  test  design  manpower,  test  stations 
and  other  resource  requirements  must  be  established  if  the  process  is  to  be  completed 
in  a timely  and  efficient  manner.  The  end-objective  of  the  plan  should  be  the 
availability  of  a full  set  of  quality  TPSs  for  weapon  systems  support  on  the  initial 
operational  capability  (IOC)  date.  This  planning  effort  can  then  be  used  by  the 
integrating  contractor  as  a basis  for  establishing  subsystem/equipment  development 
schedules  to  accommodate  TPS  acquisition  input  requirements  in  a way  which  will 
permit  a gradual  buildup  of  trained  workforce  to  sustain  the  planned  workload.  Other 
timing  elements  to  be  considered  when  developing  the  plan  may  be  the  implementation 
of  rate  tooling  and  provisions  required  for  competitive  procurement. 

b.  Organization  of  the  TPS  acquisition  process.  The  consequences  of  overly 
segmented  and  disorganized  test  program  development  efforts  in  the  past  suggest  the 
requirement  for  a defined  type  of  organization  and  workload  management  which  can  be 
specified  for  contractual  compliance.  In  general,  the  individual  test  program 
development  should  be  managed  as  a work  package  requiring  continuity  of  personnel 
assigned  as  a team  from  initial  analysis  of  test  requirements  through  program 
acceptance.  Basic  qualifications  and  experience  levels  of  team  members  can  be  found 
in  the  Program  Design  Handbook  for  Automatic  Test  Equipment  for  programs-  of 
significant  complexity.  No  such  information  has  been  developed  for  relatively  simple 
programs  or  those  developed  with  ATG  systems.  In  addition,  for  any  major  TPS 
development  contract,  few,  if  any,  contractors  have  the  requisite  number  of  qualified 
personnel  immediately  available.  Training  requirements  must  be  identified  and 
implemented  in  training  plans  to  provide  orderly  augmentation  of  the  existing  qualified 
work  force. 
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5. 1.2.3  Procurement  area.  This  activity  includes  the  mechanisms  by  which  a 
contractor  is  selected  and  the  TPS  acquisition  placed  on  contract. 


a.  Historically,  TPSs  have  been  procured  from  the  weapon  system  integration 
contractor  as  part  of  the  unpriced  line  item  for  ILS.  This  method  has  many 
disadvantages  and  most  are  typical  of  the  outcome  of  non-competitive  procurements: 
there  is  no  assurance  that  either  the  most  qualified  or  the  least  expensive  source  has 
been  found.  Of  these  two  characteristics,  the  most  important  is  the  assurance  that  the 
most  competent  contractor  has  been  selected.  The  cost  issue,  while  not  unimportant, 
is  more  difficult  to  deal  with.  No  TPS  cost  estimating  methods  have  been  developed 
which  can  be  used  readily  to  evaluate  the  realism  of  cost  quotations.  For  this  reason, 
and  because  of  the  nature  of  the  TPS  development  process  itself,  a cost  type  contract, 
awarded  on  the  basis  of  most  technically  qualified,  would  seem  to  be  the  best 
approach.  Criteria  for  proposal  evaluation  could  be  based  on  the  definition  of 
management  requirements  proposed  in  sub-paragraph  5. 1.2.2  above. 

b.  The  policy  that  TPS  development  should  be  procured  competitively  must 
take  into  consideration  other  contractual  arrangements  which  might  be  affected.  Of 
primary  importance  are  the  implications  of  such  a procurement  on  the  integration  or 
prime  contractor  who  otherwise  may  have  total  responsibility  for  ILS  development,  as 
well  as  responsibility  for  TPS  acquisition  inputs  for  other  than  GFE  avionics.  Other 
issues  which  must  also  be  resolved  are  who  prepares  the  solicitation,  conducts  the 
source  selection  and  awards  the  contract.  If  these  functions  are  to  be  performed  by 
the  government,  what  are  the  integration  contractor's  responsibilities  at  the  input  and 
output  interfaces?  Is  he  eligible  to  compete  for  the  award  under  these  circumstances? 
From  these  and  other  questions  that  suggest  themselves,  it  is  clear  that  further  study 
is  needed  to  ensure  orderly  implementation  of  a competetive  TPS  procurement  policy. 

c.  Another  procurement  consideration  requiring  investigation  is  the  use  of 
warrantee  provisions  for  correction  of  defects  and/or  maintenance  of  TPSs  after 
delivery.  These  provisions  can  be  incentivised  or  included  in  the  development  contract 
tasks.  In  either  case,  they  are  largely  dependent  on  the  development  of  valid  quality 
control  procedures  and  definition  of  defects  related  thereto. 

5.1.3  Recommendations.  Based  on  the  results  of  the  analysis  presented  in  paragraph 
5.1.2,  the  following  actions  are  recommended: 

a.  Using  material  and  structure  as  appropriate  from  the  Program  Design 
Handbook  for  Automatic  Test  Equipment,  produce  a TPS  Design  Handbook  incorporat- 
ing new  material  developed  to  satisfy  the  requirements  of  sub- paragraphs  5.1.2.1b 
and  c and  5.1.2.2b.  As  a separate  but  related  action,  establish  and  initiate  a task  to 
develop  valid  and  applicable  quality  control  procedures/techniques  for  incorporation  in 
the  TPS  Design  Handbook  and  TPS  requirements  specifications  applicable  to  TPS 
procurement  contracts  (5.1.2.1c). 
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b.  Revise  AR-9B  and  coordinate  as  a Navy-wide  TPS  requirements  specifica- 
tion (3.1.2.1b,  c,  d). 

c.  Define  and  initiate  the  following  tasks: 

1.  Develop  policy  guidelines  for  application  of  CAD  systems  in  test 
program  design  based  on  an  investigation  of  current  capabilities 
(5.1.2.1a). 

2.  Develop  detailed  requirements  for  a TPS  Acquisition  Plan  to  be 
applied  to  weapon  systems  contracts  (5.1.2.2a). 

3.  Develop  organization,  staffing  and  workload  management  guidelines 
for  the  TPS  development  process  (5.1.2.2b). 

4.  Investigate  the  feasibility  and  requirements  for  implementation  of 
cost-type,  competitive  contracts  for  TPS  procurements,  including 
optimum  warrantee  provisions.  Prepare  procurement  policy  state- 
ments as  appropriate. 

d.  Based  on  these  results,  direct  the  preparation  of  a TPS  Acquisition  Guide  to 
present  at  an  appropriate  level  of  detail  the  key  events  and  requirements  of  the  TPS 
acquisition  process  in  a time-phased,  input-output  relationship  to  the  acquisition 
process  of  prime  weapon  systems. 

5.2  ATE  planning  requirements. 

5.2.1  Overview.  The  acquisition  and  implementation  of  a viable  ATE  support  concept 
requires  certain  up-front  planning  requirements  which  must  be  addressed  early  during 
weapon  systems  development.  The  Acquisition  Planning  Guide  for  Automatic  Test, 
Monitoring  and  Diagnostic  Systems  and  Equipment  issued  under  NAVMATINST  3960.9 
attempts  to  do  just  that.  The  material  contained  in  this  guide  is  structured  to  assist 
program  managers,  ATE  acquisition  managers,  and  others  associated  with  the 
acquisition  of  Navy  hardware,  both  in  government  and  industry,  in  relating  acquisition 
planning  requirements  to  activities  undertaken  during  each  phase  of  the  syrtem 
acquisition  process  as  prescribed  in  5ECNAVINST  5000.1.  However,  as  a result  of 
additional  inputs  received  from  selected  project  and  field  data  surveys,  correspondence 
with  cognizant  tri-service  and  SYSCOM  ATE  personnel,  and  an  analysis  of  the 
functional  flow  of  acquisition  and  developmental  activities  in  Figure  1-1  of  this  guide, 
it  has  been  determined  that  additional  refinement  of  this  document  is  lequired.  Policy 
and  procedural  requirements  and  guidance  are  required  in  the  following  areas: 


o ATE  Selection 

o ATE  Workload  Analysis 

o Testability  Requirements. 


A brief  discussion  of  the  issues  involved  in  each 
following  sub-paragraphs. 


of  these  topics  is  provided  in  the 
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5.2.2  ATE  selection. 

5.2.2. 1 Problem.  A rational,  straightforward,  and  cost-effective  methodology  for 
selecting  ATE  has  been  sought  by  the  support  community  for  some  time.  During  the 

' late  sixties,  the  demand  for  this  methodology  was  not  critical  because  relatively  few 

different  types  of  ATE  systems  were  available.  ATE  during  that  time  frame  consisted 
primarily  of  large-scale  general  purpose  types  (i.e.,  VAST)  or  custom  versions  (i.e., 
peculiar  ground  support  equipment).  It  wasn't  until  the  early  seventies  that  equipment 
manufacturers  began  to  comprehend  the  potential  of  the  ATE  market  and  responded  by 
introducing  equipments  with  hardware  and  software  attributes  aimed  at  particular 
voids  in  the  marketplace.  In  some  instances,  equipment  manufacturers  introduced 
equipments  similar  to  their  competitors'  for  the  primary  purpose  of  getting  into  the 

' market  as  quickly  as  possible.  In  other  instances,  equipment  manufacturers  introduced 

ATE  systems  which  were  based  on  technology/concepts  which  exemplified  corporate 
strengths  and  long  range  marketing  goals  rather  than  the  real  needs  of  potential  users. 

Buzz  words  such  as  "rack  and  stack"  and  "third  generation"  came  into  use  and  split 
ATE  manufacturers  and  users  into  diverse  camps.  This  condition  confused  many 
potential  ATE  users  and  weapon  systems  support  managers  in  particular.  Basically, 
these  managers  wanted  to  know  how  they  could  objectively  answer  two  questions: 

a.  When  is  automatic  test  equipment  justified? 

b.  How  do  you  go  about  deciding  what  ATE  to  select? 

5.2. 2. 2 Analysis.  Both  the  Navy  and  Air  Force  have  attacked  these  problems  in  one 
manner  or  another  but  current  guidance  and  procedures/tools  to  answer  these  two 
bottom  line  questions  is  still  not  adequate.  With  respect  to  the  second  question, 
various  data  banks  over  the  past  few  years  have  been  under  development  by  the 
Air  Force  and  Navy  for  the  express  purpose  of  providing  an  objective  means  of 
selecting  ATE.  Although  there  is  no  direct  relationship  or  interface  among  these  data 
banks,  there  is  a certain  amount  of  overlap  of  information  among  them.  The  data 
banks  are  structured  differently  and  each  was  designed  to  perform  a separate  and 
distinct  function.  All  of  the  data  banks  have  one  common  goal:  to  identify  existing 
support  equipment  of  one  type  or  another.  The  Naval  Air  Engineering  Center  (NAEC) 
in  Lakehurst,  New  Jersey,  instituted  a pin  by  pin  matching  program  in  a data  bank  to 
match  ATE  with  UUTs.  The  Air  Force's  San  Antonio  Air  Logistics  Center  has  also 
developed  an  ATE  data  bank  for  the  express  purpose  of  selecting  ATE  based  upon  the 
test  requirements  of  UUTs.  The  Air  Force's  data  bank  differs  from  the  Navy's  data 
bank  in  that  it  provides  a functional  requirements  analysis  of  UUTs  versus  ATE  in  lieu 
of  a detailed  pin  for  pin  analysis.  Both  data  banks  have  their  shortcomings.  The 
Navy's  data  bank  contains  parametric  data  on  approximately  a dozen  ATE  systems  and 
requires  extensive  upgrading  to  reflect  what  is  currently  in  the  Navy's  inventory  and 
what  is  currently  available  in  the  commercial  marketplace.  In  addition,  the  input 
coding  requirements  for  UUTs  (pin  information)  have  significant  cost  implications. 

The  Air  Force  approach,  using  a functional  matching  concept,  incurs  some  technical 
risk  in  the  matching  process  because  of  the  general  nature  of  UUT  test  requirements 
data. 
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The  industry's  ATE  project  for  the  Navy  conducted  a study  (Task  E)  of  various 
ATE  data  banks.  The  results  of  this  study  are  documented  in  Volume  III  (Management 
Considerations:  Conclusions  and  Recommendations)  of  the  Report  of  Industry  Ad  Hoc 
Automatic  Test  Equipment  Project  for  the  Navy.  This  report  reflects  the  need  for 
objective  selection  of  ATE  via  a computerized  data  bank.  However,  the  report  also 
recognized  that  existing  ATE  data  banks  are  not  fully  operational  and/or  loaded  with 
current  information. 

Based  upon  analysis  of  survey  data  gathered,  Task  Group  E generated  several 
conclusions  and  recommendations  pertaining  to  data  banks.  A good  majority  of  these 
recommendations  and  conclusions  pertained  to  the  personnel  and  data  requirement 
aspects  of  ATE  data  banks.  In  terms  of  proposed  policy,  the  ad  hoc  group  made  the 
following  recommendation: 

After  the  successful  implementation  of  the  above 
recommendations,  it  is  further  recommended  that  all  contracts 
which  are  ATE  impacted,  to  the  extent  of  $100,000  or  more,  be 
made  to  contain  contractual  provisions  that  make  mandatory 
the  use  of  ATE  data  banks  in  such  matters  as  ATE  selection  or 
design,  in  "unit  under  test"  test  program  development,  and  in 
related  research  and  development  efforts.  Costs  incurred  in 
using  the  data  bank  should  be  direct  allowable  charges  against 
the  contract. 

This  recommendation  satisfies  long  range  rather  than  short  range  policy  needs  in 
the  data  bank  area. 

With  respect  to  selecting  ATE  objectively  by  matching  ATE  characteristics  with 
UUT  test  requirements,  the  Air  Force  made  the  following  statement  at  the  recent  (15- 
16  June  1977)  tri-service  briefing  on  automatic  testing  in  its  paper.  The  San  Antonio 
Air  Logistics  Center  Automatic  Test  Equipment  Data  Bank: 

At  the  present  time,  there  is  no  standard  vehicle  to  apply  the 
requirement  for  use  of  the  data  bank  to  new  contracts.  We 
have  recognized  for  quite  some  time,  the  need  for  a data  item 
description  (DID)  requiring  a contractor  to  deliver  properly 
coded  test  requirements  for  screening  against  the  data  bank. 

However,  action  was  delayed  until  we  progressed  through  the 
conceptual  and  development  phases  and  were  assured  the  data 
bank  was  a capable  screening  tool  and  would  be  utilized. 

This  approach  is  directly  opposite  to  that  taken  by  the  Navy.  By  NAVMAT 
Instruction  5230.8  (14  November  1974),  the  Deputy  Chief  of  Naval  Material  made 
mandatory  the  use  of  data  banks.  Despite  this  instruction,  inquiries  made  relative  to 
ATE  selection  process  from  November  1974  through  23  July  1976  were  minimal  to 
nonexistent  at  these  data  banks. 
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In  summary  , it  appears  that  an  alternative  short-term  methodology  and  guidance 
to  the  ATE  selection  process  independent  of  data  bank  utilization  is  required  in  support 
of  Navy  programs. 

5.2.2. 3 Recommendations.  In  the  light  of  the  previous  discussion,  immediate  guidance 
and  methodology  pertaining  to  the  ATE  selection  process  is  needed.  It  is  recommended 
that  this  guidance  be  provided  in  the  Acquisition  Planning  Guide  for  Automatic  Test, 
Monitoring  and  Diagnostic  Systems  and  Equipment.  The  Annex  of  Appendix  1 describes 
a proposed  methodology  which  may  be  integrated  into  the  ATE  Acquisition  Guide  to 
provide  short  term  guidance  to  acquisition  managers. 

It  is  also  recommended  that  data  band  I/O  specifications,  access  turnaround 
times,  currency  of  information,  and  cost  data  be  published  and  validated  prior  to 
requiring  compulsory  utilization  of  these  resources  via  policy  directives. 

5.2.3  ATE  workload  analysis  and  justification. 

5.2.3. 1 Problem.  No  standardized  Navy  policy,  procedures,  or  operational 
methodology  exists  for  justifying  ATE,  or  for  performing  cost  benefits/trade-off  and 
workload  analyses.  As  a result,  proliferation  and  the  uneconomical  utilization  of 
support  resources  results  because  no  uniform  means  of  measuring  or  ascertaining  the 
effectiveness  of  one  support  alternative  over  another  exists. 

5. 2.3. 2 Analysis.  A number  of  procedurally  oriented  tools  currently  exist  which  enable 
project  managers  to  objectively  conduct  cost  benefits/trade-off  analyses  of  ATE  with 
other  suppxjrt  alternatives.  The  Air  Force's  MIL-STD-1513,  Trade  Studies  for  the 
Selection  of  Avionic  Support  Systems,  Criteria  for,  and  the  Navy's  Test  Equipment 
Effectiveness  Model  (TEEM)  are  two  tools  available  to  project  managers.  However, 
they  are  not  operationally  suitable  for  Navy  application  at  this  time.  The  TEEM  model 
provides  the  ILS  and  acquisition  manager  a tool  to  evaluate  what  type  of  test 
equipment  to  place  at  each  repair  site.  Results  indicate  placement  which  will  achieve 
the  most  effective  support  of  a weapon  system.  The  model  evaluates  results  of 
assigning  one  of  three  tester  categories:  built-in-test,  off-line  manual  test  equipment 
(MTE)  and  automatic  test  equipment  to  a specific  site.  The  site  locations  are 
identified  as  organizational,  intermediate  or  depot. 

TEEM  is  intended  for  use  early  in  the  development  phases  of  a system  acquisition 
process.  To  this  end,  the  model  has  been  maintained  relatively  simple  with  limited 
input  requirements  commensurate  with  limited  data  availability  and  accessibility. 

The  output  from  the  model  provides  reports  on  availability,  quantities,  cost 
breakdown  and  sensitivity  analyses.  The  availability  report  indicates  a measure  of 
performance  of  the  test  equipment  by  showing  availability  of  the  prime  system  for 
each  identified  support  method.  The  quantities  report  is  an  indication  of  test 
equipment  utilization.  It  provides  information  concerning  numbers  of  test  stations 
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needed  for  each  tester  type,  number  of  trained  personnel  to  operate  testers,  wait  time 
in  queue  and  number  of  items  in  queue.  The  cost  breakdown  provides  a printout  of 
both  recurring  and  non-recurring  costs.  Recurring  costs  include  areas  such  as 
' transportation,  labor,  space  and  test  equipment  support.  Non-recurring  costs  address 

acquisition  of  test  equipment  and  other  on-time  costs.  The  sensitivity  analysis  report 
provides  information  on  changes  experienced  in  number  of  tests  required,  prime  system 
spares  requirements  and  probability  of  testers  being  utilized  for  changes  in  mean  time 
between  removals  of  assemblies  the  tester  checks. 

It  is  important  to  note  that  the  model  does  not  satisfy  the  requirements  for  a 
level  of  repair  (LORA)  or  life  cycle  cost  (LCC)  analysis.  It  is  instead  a supplement  to 
those  analyses.  Generalized  repair  versus  discard  decisions,  for  instance,  should  be 
made  prior  to  using  the  model.  If  a tentative  LORA  decision  has  not  been  made,  the 
analyst  must  make  several  runs  of  the  model,  varying  the  test  scenario  to  cover  the 
most  probable  LORA  decisions.  Also,  a complete  LCC  analysis  must  consider  cost 
categories  which  the  model  does  not  include.  In  summary,  the  model  is  basically  a 
sensitivity  analysis  tool  which  can  be  utilized  in  selecting  support  alternatives. 

The  TEEM  model  contains  certain  technical  deficiencies  (Appendix  G)  which 
were  identified  during  the  early  stages  of  the  policies  and  procedures  effort  and  are 
now  being  rectified. 

The  purpose  of  MIL-STD-1513  is  to  effect  the  selection  of  a test  support 
configuration  (automatic,  manual,  or  a combination)  for  avionic  systems  that  can  be 
optimally  utilized  at  specific  levels  of  maintenance.  This  selection  process  is  based 
upon  performing  trade  studies  to  determine  the  optimal  test  configuration.  The 
standard  requires  that  the  following  studies  be  conducted  as  part  of  the  trade  studies: 

o Preliminary  Engineering  Analyses  and  Decision  Studies 
UUT  Identification 

' - UUT  Functional  Characteristics 

Test  Equipment  Component  Identification 
Interface  Hardware  Identification 

Test  Set  Component  Cost  Determination  (recurring  and  non- 
recurring) 

o UUT  versus  Test  Requirements  Matrix  Analyses 

o Workload  Analyses 

Determination  of  UUT  Arrivals 

Determination  of  Off-line  Test  Station  Test  Times 

o Conceptual  Synthesis  of  ATE  and  MTE  Configurations 

Application  of  Effectiveness  Criteria 

Queueing  Analyses 

Test  Station  Optimization  Analyses 
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o Life  Cycle  Cost  Analyses 

o Cost  Effective  Analyses 

o Optimal  Test-Support  Configuration  Analyses 

o Optimum  Repair  Level  Analysis  (ORLA). 

MlL-STD-1513  falls  short  in  detailing  procedurally  as  to  how  one  arrives  at  the 
optimum  ATE  support  vehicle  to  support  "n"  UUTs.  It  is  generally  recognized  in  the 
ATE  community  that  the  optimization  procedure  is  a complex  interactive  process. 
MIL-STD-1513  does  not  impose  any  specific  analysis  or  methods  to  arrive  at  an 
optimum  ATE  configuration.  It  only  requires  that  all  rationale  be  documented  and 
submitted  to  justify  the  configuration  and  optimization  techniques.  Thus,  a mystique 
is  cast  about  the  process  by  which  one  optimizes  or  selects  an  ATE  support  vehicle. 
Unless  this  mystique  is  dispelled,  project  personnel  will  not  carry  out  life  cycle  cost 
trade-offs  between  general  purpose  electronic  test  equipment  (CRETE)  and  ATE,  and 
test  equipment  proliferation  and  support  costs  will  continue  to  escalate.  In  addition, 
the  methodology  utilized  for  predicting  UUT  workload  is  based  upon  workload  derived 
from  a typical  air  wing  operational  scenario.  This  scenario  assumes  a uniform  arrival 
rate  of  UUTs  over  a given  period  of  time  and  does  not  address  the  various  real  life 
workload  scenarios  which  are  peculiar  to  ships  interfacing  with  depots,  intermediate 
maintenance  activities  (IMA),  and  tenders  where  arrival  and  service  rates  of  support 
equipment  are  time  dependent  such  as  in; 

a.  A "rush  hour"  scenario  of  "n"  UUTs  to  be  serviced  in  a finite  period  of  time 
"t"  to  meet  specified  mission  availability  requirements. 

b.  A priority  queueing  scenario  where  "n"  UUTs  arrive  after  "x"  UUTs  are  in 
the  process  of  being  serviced  by  a tester  and  warrant  immediate  service. 
The  probability  and  the  impact,  from  a readiness  point  of  view,  on  the 
average  wait  in  the  queue  for  each  of  the  "x"  UUTs  is  not  addressed  in  MIL- 
STD-1513. 

In  addition,  MIL-STD-1513  does  not  require  that  life  cycle  cost  trade-offs  of  BIT 
(another  support  equipment  alternative)  be  conducted  along  with  manual  test 
equipment  and  ATE  alternatives. 

5.2. 3. 3 Recommendations.  There  is  a need  in  the  support  community  for  both 
computerized  and  non-computerized  methodology  for  performing  cost  benefits  and 
workload  analyses.  As  stated  previously,  an  analysis  and  recommendations  pertaining 
to  the  computerized  TEEM  model  can  be  found  in  Appendix  G.  With  respect  to  the 
non-computerized  approach  (MIL-STD-1513),  the  following  recommendations  are  made: 

a.  Research  and  update  MIL-STD-1513  queueing  and  workload  analysis 
equations  to  reflect  both  air  and  sea  applications  with  particular  emphasis 
on  rush-hour  queueing  and  priority  queueing. 

b.  Provide  more  guidance  in  MIL-STD-1513  as  to  the  methodology  by  which  an 
optimum  support  vehicle  is  selected. 
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c.  Provide  guidance  in  MIL-STD-1513  as  to  how  life  cycle  costs  for  Navy 
programs  are  to  be  derived.  MIL-STD-1513  presently  references  Air  Force 
documents  to  perform  these  calculations. 

d.  Include  BIT  as  a support  alternative  which  must  be  traded  off  with  ATE  and 
MTE  at  a specified  level  of  maintenance. 

5.2.4  Testability  requirements. 

5.2.4. 1 Statement  of  problem.  The  lack  of  BIT,  system  readiness  information,  and 
amenability  to  off-line  testing  of  weapon  systems  has  caused  severe  maintenance 
problems  in  the  Fleet.  A standard  which  will  require  that  new  prime  equipment 
contain  these  features  is  needed.  The  Report  on  Navy  Issues  Concerning  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment  identified  those  issues  as 
problem  numbers  17  (lack  of  effective  built-in-test),  18  (lack  of  command  informa- 
tion), and  20  (items  not  amenable  to  test). 

5. 2. 4. 2 Analysis.  The  specification  of  BIT,  test  point,  and  interface  standards  is  a 
means  of  imposing  testability  requirements  on  prime  equipment.  During  the  past 
decade,  NAVAIR  has  imposed  such  requirements  on  prime  equipment  manufacturers  by 
imposing  AR-8  (general  requirements  for  versatile  avionic  shop  test  system/avionics 
systems  compatibility)  and  AR-IO  (general  requirements  for  maintainability  of  avionics 
equipments  and  systems).  With  respect  to  the  ship  side  of  the  house,  one  of  the  first 
attempts  at  establishing  testability  requirements  on  prime  equipments  design  was  the 
generation  of  MIL-STD-1326  (test  points,  test  point  selection  and  interface  require- 
ments for  equipments  monitored  by  shipboard  on-line  ATE). 

This  standard  establishes  the  requirements  for  providing  test  points  in  prime 
equipments  for  monitoring  by  on-line  automatic  test  equipment.  It  provides  criteria 
for  guidance  in  optimum  test  point  selection.  It  also  defines  certain  interface  and  data 
requirements,  a system  of  test  point  data  generation,  and  procedures  for  submission  of 
data  disclosing  the  selection  of  these  test  points.  It  is  the  intent  of  this  standard  to 
specify  requirements  to  achieve  the  following  testability  objectives: 

a.  The  optimum  selection  and  placement  of  test  points  to: 

Continuously  monitor  the  performance  of  prime  equipment. 

Detect  the  existence  of  a failure. 

Facilitate  rapid  isolation  of  a failure  to  the  line  replaceable  unit  to 
effect  repair  by  substitution  of  a spare,  performance  of  realignment, 
etc. 

b.  The  planning  and  development  of  an  adequate  level  of  test  logic  design  for 
the  prime  equipment  in  order  that  the  ATE  can  be  programmed  to  provide 
optimum  monitoring  of  sensor  outputs,  and  to  ensure  timely  delivery  of  the 
end  article  and  all  of  the  required  test  point  information. 

c.  The  specification  of  the  types  of  test  point  signals  and  their  associated 
electrical  characteristics  which  must  be  provided  for  ATE  monitoring. 
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Figure  2 is  a pictorial  rendering  of  the  effect  MIL-STD-1326  has  had  on  prime 
equipment  interface  design. 

As  noted  earlier,  the  test  point  interface  is  predominantly  analog  and  is 
unidirectional  in  nature.  Also  note  that  for  some  test  point  signals,  signal  conditioning 
sensors  had  to  be  built  into  the  prime  equipment  in  order  to  make  compatible  the 
characteristics  of  these  test  point  signals  with  those  specified  in  this  standard. 

MIL-STD-1326  has  certain  distinct  advantages  and  disadvantages.  The  primary 
advantages  are: 

a.  Placing  a bound  on  the  types  of  measurements  and  electrical  character- 
istics of  test  point  signals  reduces  the  need  for  overly  complex  off-line 
ATE. 

b.  The  recommended  test  logic  approves  of  utilizing  the  specified  test  points 
is  a straightforward  approach  of  sequentially  monitoring  a separate  set  of 
test  points  for  each  unique  mode  of  prime  equipment  operation. 

The  disadvantages  or  shortcomings  of  MIL-STD-1326  appear  to  far  outweigh  its 
apparent  advantages.  Some  of  the  primary  disadvantages  are: 

a.  The  standard  addresses  the  requirements  for  establishing  standard  analog 
test  point  interfaces  between  prime  equipments  and  specialized  on-line 
testers.  Digital  test  interface  standards  are  not  addressed  in  this  standard. 

b.  The  standard  specifies  a boundary  on  the  electrical  characteristics  of  test 
points  and  the  types  of  measurement  capability  available  for  their 
monitoring  which  increases  the  complexity  and  cost  overhead  of  prime 
equipments. 

c.  The  recommended  sequential  test  logic  concept  proposed  by  MIL-STD-1326 
is  for  the  most  part  more  compatible  with  analog/RF  equipments  than  for 
digital  equipments. 

d.  The  standard  contains  no  reference  to  or  requirement  for  BIT  in  prime 
equipment. 

e.  The  standard  is  not  compatible  with  current  prime  equipment  technology 
and  recent  developments  in  shipboard  data  multiplex  systems. 

f.  The  standard  does  not  provide  to  the  subsystem  designer  guidance  in 
establishing  how  test  point  data  is  to  be  formatted  and  utilized  for 
operational  readiness  monitoring  system  (ORMS)  application  at  the  ships 
level. 

In  light  of  these  disadvantages,  the  ships  community  embarked  on  an  effort  to 
update  MIL-STD-1326  to  an  "A"  revision  in  order  to  rectify  its  shortcomings.  The 
requirement  for  upgrading  MIL-STD-1326  was  identified  during  the  Navy's  ATE 
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Interface  Requirements  Imposed  on  Prime  Equipments 
by  MIL-STD  1326A. 


Some  of  the  shortcomings  of  MIL-STD-1326A  are  as  follows: 

a.  The  standard  does  not  specify  analog  interface  requirements  between  prime 
equipment  and  on-line  ATE.  That  is,  no  electrical  bounds  in  terms  of 
voltage  and  frequency  are  specified  for  test  point  and  performance 
monitoring  signals. 

b.  The  standard  does  not  specify  the  characteristics  and  measurement 
capabilities  of  the  on-line  ATE  that  the  prime  equipment  will  interface 
with. 

c.  As  in  MlL-STD-1326,  this  standard  does  not  provide  guidance  to  the 
subsystem  designer  in  establishing  how  test  point  data  can  be  formatted 
and  utilized  for  OR  MS  application  at  the  ships  level. 

d.  MIL-STD-1326A,  despite  its  functional  attributes,  is  another  standard  in  a 
maze  of  BIT/test  point/data  bus  standards. 

Figure  k amplifies  this  Icist  point.  This  figure  depicts  in  matrix  format  eleven 
major  BIT/test  point/interface  bus  related  documents.  Appendix  F provides  in 
summary  format  the  scope,  contents,  and  general  information  on  each  of  these 
documents  with  respect  to  Figure  4.  Note  that  only  the  proposed  MIL-STD-1326A 
provides  BIT,  test  point,  and  data  bus  specification  requirements  in  one  document. 
However,  perusal  of  this  matrix  quickly  brings  two  obvious  questions  to  mind: 

a.  How  does  MIL-STD-1326A  interface  or  overlap  with  these  specifications  or 
documents? 

b.  In  the  light  of  the  existence  of  these  documents,  is  the  need  for  a new 
specification  really  warranted? 

Figure  5 depicts  the  interrelationships  of  the  documents  noted  on  the  previous 
matrix.  All  documents  drawn  in  dotted  ellipses  with  a star  (*)  internal  to  them  are 
documents  which  are  evolving  and  have  not  yet  been  officially  released.  All  other 
documents  drawn  in  dotted  ellipses  without  a star  (*)  internal  to  them  are  released 
documents  undergoing  revision  at  this  time.  Arrows  interconnecting  ellipses  in  an 
oblique  or  horizontal  manner  indicate  a casual  connection  between  these  documents. 
Arrows  emanating  from  one  ellipse  perpendicular  to  another  ellipse  in  a top-down 
manner  indicate  a strong  connection  between  documents  (i.e.,  MlL-STD-1399  refer- 
ences MlL-STD-1326  for  guidance  in  shipboard  test  point  design).  A dotted  arrow 
indicates  that  an  implied  relationship  should/could  exist  between  two  documents,  one 
(or  both)  of  which  is  yet  to  be  released. 

An  analysis  of  Figure  5 indicates  that  coordination  in  effecting  data  bus 
standardization  is  being  worked  on  across  Navy  SYSCOM  and  service  lines.  This  area 
appears  to  be  receiving  proper  attention  with  respect  to  the  establishment  of  common 
data  bus  standardization  requirements. 
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However,  a state  of  flux  appears  to  exist  at  the  moment.  This  situation  is  due 
primarily  to  the  problems  incurred  in  integrating  individual  data  bus  requirements  from 
different  service  branches  into  a universal  document.  Coordination  of  this  effort  has 
received  attention  at  high  management  levels  within  the  Navy.  The  adoption  of  a 
universal  data  bus  standard  would  achieve  cross  platform  (aircraft/ship)  interface 
standardization  both  within  the  U.S.  and  throughout  NATO. 

The  future  effect  of  this  effort  on  ATE  and  testing  technology  would  be 
beneficial  in  that  the  communications  data  bus  protocol  adopted  could  also  be  utilized 
as  an  ATE/UUT  communication  standard.  Such  a standard  could  effect  a reduction  in 
the  number  of  unique  hardware/software  data  communication  interfaces  required  for 
automatic  test  systems. 

With  respect  to  test  point  and  BIT  requirements,  Figure  5 indicates  that  no  solid 
or  dotted  horizontal  connection  appears  across  SYSCOM  or  service  lines.  Figure  5 also 
indicates  that  there  is  an  absence  of  a BIT  related  standard  in  the  ships  community  for 
prime  equipment.  Note  that  the  air  community  covers  this  void  in  its  arena  via  AR-10 
which  is  slated  to  become  a MIL-STD  in  the  near  future.  Figure  5 demonstrates  that 
this  situation  coupled  with  a rather  specialized  and  outmoded  (i.e.,  analog  only) 
interface  specification  (i.e.,  MIL-STD- 1326)  and  the  new  felt  presence  of  an  evolving 
shipboard  signal  interface  specification  makes  a strong  case  for  generating  a new 
BIT/test  point/interface  related  specification  for  prime  shipboard  equipments.  How- 
ever, perusal  of  Figure  5 makes  one  ask  the  question  "Why  generate  another 
specification  in  the  wake  of  the  plethora  of  documents  available  to  use?"  Better  yet, 
one  might  ask  "Why  not  come  up  with  a universal  (i.e.,  standard)  BIT/test  point/data 
bus  standard?"  For  the  uninitiated,  such  an  approach  may  seem  sensible.  However, 
this  approach  is  not  compatible  with  the  lessons  learned  from  prior  association  with 
such  tasks. 

Perhaps  the  comments  made  by  industry  under  Task  1 (Specification  Review)  of 
the  Ad  Hoc  project  for  the  Navy  best  exemplifies  this  point: 

...wide  diversity  of  devices  and  components  now  supported  by 
automatic  test  equipment  makes  very  difficult  and  perhaps 
undesirable  the  broad  standardizing  of  test  points,  measurement 
data  parameters,  and  values  thereof. 

5. 2. 4. 3 Recommendations.  With  respect  to  the  need  for  a new  ship  testability 

standard,  the  following  recommendations  are  made: 

a.  A review  of  the  proposed  MIL-STD- 1326 A in  unison  with  other  BIT/test 
point/interface  bus  related  documents  indicates  that  such  a document  is 
needed  in  the  ships  community.  However,  it  is  recommended  that  the 
proposed  MIL-STD- 1326A  as  it  presently  stands  not  be  the  vehicle  by  which 
this  standardization  process  be  effected.  The  proposed  standard  as  it 
stands  now  is  in  conflict  with  other  testability  documents.  It  is 
recommended  that  the  revision  of  MIL-STD- 1326  continue  utilizing  the 
draft  of  MIL-STD- 1326 A and  be  coordinated  with  existing  standards  and 
Navy  project  efforts  (i.e.,  ORMS  and  SDMS). 
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In  the  revised  standard,  establish  the  criteria  which  should  be  used  in 
subsystem  design  for  determining  the  output  data  requirements  for  OR  MS. 
That  is,  establish  how  to  use  existing  test  points  and  outputs  to  define  the 
various  levels  of  equipment  operational  readiness. 

In  the  revised  standard,  ensure  that  the  electrical  characteristics  of  all  test 
point  interface  signals  are  specified  to  be  compatible  with  OR  MS  and 
SDMS. 


5.3  Test/repair  documentation. 

5.3.1  Overview.  Technical  documentation  in  support  of  ATE  TPS  development  and 
operational  deployment  has  in  the  past  had  serious  deficiencies.  These  deficiencies, 
addressed  in  the  Marcy  Report,  primarily  deal  with  the  content  and  the  untimely 
delivery  of  technical  documentation  to  operational  personnel. 

Since  1972,  module  schematics,  parts  lists,  and  training  have  usually  not  been 
procured  with  NAVSEA  equipments  in  order  to  minimize  initial  system  acquisition 
costs.  NAVAIR,  who  earlier  tried  this  approach,  found  it  counterproductive  and  was 
forced  to  resume  procurement  of  module  schematics,  parts  lists,  training,  and  other 
test/repair  documentation  with  each  equipment  procurement.  Therefore,  modules 
procured  for  NAVSEA  equipment  after  1972  are  usually  coded  for  depot  (or 
manufacturer)  repair  or  throwaway  because  the  documentation  needed  to  effect  Navy 
organic  repair/test  of  modules/printed  circuit  boards  (PCBs)  is  not  available.  This 
support  posture  has  seriously  stressed  Operations  and  Maintenance,  Navy(0(5cMN) 
funding  for  operational  systems  and  has  had  a detrimental  effect  on  the  life  cycle 
costs  for  operational  systems. 

A policy  requiring  project  managers  to  consider  the  cost/benefits  of  procuring 
test/repair  documentation  during  initial  system  acquisition  phase  is  required. 
Documentation  required  in  support  of  ATE  may  be  classified  into  three  general 
categories  as  follows: 

o UUT  Test/Repair  Documentation 

o ATE  System  Documentation 

o TPS  Documentation. 

A brief  description  of  each  of  these  documentation  elements  follows  with  a statement 
of  problem  as  it  relates  to  each  element,  analysis  undertaken,  and  proposed 
recommendations  to  rectify  the  anomalies  identified. 

5.3.2  UUT  test/repair  documentation. 

5.3.2. 1 Problem.  The  data  items  necessary  to  support  UUT  test/repair  are  untimely, 
unclear,  disorganized,  non-standardized  and  inadequate  for  timely  troubleshooting 
(reference  Marcy  Report  13  February  1976). 
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5. 3. 2.2  Analysis.  The  key  issue  concerning  the  preparation,  quality  and  delivery  of 
adequate  test/repair  documentation  is  cost. 

' Acquisition  managers  argue  that  to  require  them  to  procure  TRD  specified 

documentation  will  inflate  their  initial  weapon  system  acquisition  costs.  But  the  fact 
of  the  matter  is  that  the  cost  and  benefits  derived  from  acquiring  source 
documentation  must  be  traded  off  over  the  life  cycle  of  a weapon  system  program. 
That  is,  the  initial  investment  "P"  in  test/repair  assets  made  in  year  one  of  system 
acquisition  will  ultimately  result  in  an  operational  savings  of  "S"  dollars  per  year.  The 
initial  investment  "P"  consists  of  the  following: 

P = $ TRD  Acquisition  Costs  + $ Test  Program  Set  Acquisition  Costs  + $ ATE 

Acquisition  Costs  + $ Recurring  Costs 

The  savings  "S"  is  the  reduction  in  0<5cMN  funds  brought  about  by  effecting 
organic  off-line  support  for  UUTs.  The  savings  "S"  consists  of  the  following: 

$ 5 = Contractor  Support  Costs  + Pipeline  Costs  + Inflated  Logistics  Support  + 
Contractor  Depot  Repair  Costs 

Thus,  it  can  be  seen  that  the  cost/benefits  accrued  by  procuring  test/repair 
documentation  are  often  difficult  to  quantify  since  the  rate  of  return  on  investment  or 
operational  savings  derived  from  it  is  a function  of  the  investment  in  and  application 
of  other  capital  investment  assets. 

A vehicle  currently  exists  to  specify  documentation  type,  format  and  quality  of 
data  to  be  procured  in  support  of  repairables.  This  major  baseline  document  is  the  test 
requirements  document. 

This  document  consists  of  the  primary  specification  and  four  appendices.  The 
purpose  of  the  document  and  each  appendix  is  summarized  and  provided  as 
Attachment  1. 

The  data  elements  which  are  defined  by  the  TRD,  will  uniformly  request  the 
same  elements  provided  by  the  contractor.  Past  experience  shows,  however,  that 
while  the  same  specifications  are  used  for  drawings,  test  documents,  etc.  (i.e.,  MIL-D- 
1000),  the  end  products  often  vary  widely  in  clarity,  information  provided  and  the 
format  which  presents  the  required  information. 

A review  of  NAVAIR  Instructions  4720.4  and  5600. 20A  provides  a detailed  policy 
and  procedure  for  dissemination/promulgation  of  documentation  and  change  informa- 
tion. However,  this  distribution  scheme  is  based  on  the  premise  that  the  original 
documentation  has  been  procured  in  a timely,  efficient,  cost  effective  manner.  These 
instructions  do  not  establish  where  in  the  procurement  cycle  to  develop  the  necessary 
documentation. 
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5. 3. 2. 3 Recommendations.  After  a review  of  policy  and  procedure  concerning 
documentation  as  well  as  consideration  of  the  commentary  provided  in  the  Marcy 

' Report  and  the  NAVAIR  ATE  documentation  meeting  (26  February  1976),  the  following 

recommendations  are  made: 

a.  Establish  a policy  requiring  that  test/repair  documentation  for  operational 
systems/equipments  be  acquired  based  on  the  maintenance  concept 
adopted. 

b.  Establish  within  this  policy  that  test/ repair  documentation  be  procured 
during  system  acquisition.  OPN  funds  rather  than  0<5cMN  funds,  must  be 
identified  for  documentation. 

c.  Establish  a review  and  revision  of  MlL-D-1000  to  assure  uniform  quality  of 
data  delivery.  A procedure  to  improve  drawing  preparation  and  quality, 
which  is  included  in  the  procurement  contract,  is  required  if  MIL-D-1000 
update  is  not  practicable  . 

d.  Establish  a policy  that  UUT  documentation  be  deliverable  concurrent  with 
system/equipment  deployment. 

e.  Establish  a policy  which  will  identify  the  following  as  the  minimum  data 
requirements: 

Functional  signal  flow  diagrams 

Maintenance  manual  type  document  for  the  WRA  d:  SRAs. 

5.3.3  ATE  system  documentation. 

5.3.3. 1 Statement  of  the  problem.  In  consonance  with  the  total  system  procurement 
concept,  the  ATE  selected  to  support  the  UUT  will  have  been  established.  In  many 

cases,  the  documentation  necessary  to  support  the  ATE  test  tool  is  in  a state  which  i 

does  not  allow  adequate  use  of  the  tool.  The  ATE  becomes  a mystery  machine,  with 
its  own  set  of  anomalies  and  maintenance  problems. 

The  lack  of  adequate  uniform  documentation  and  information  results  in  another 
element  which  must  be  repaired,  maintained  and  processed  through  the  repair  pipeline. 

Downtime  in  the  ATE,  which  becomes  extended  due  to  lack  of  documentation,  causes 
increasing  queues  of  prime  equipments  which  become  backed  up  awaiting  test. 

5.3. 3. 2 Analysis.  ATE  systems  which  reflect  new  developments  have  the  same 
inherent  problems  as  UUT  development  programs.  The  cost  factors  of  initial 
documentation  procurements  coupled  with  hardware  procurements  tend  to  convince 
the  procuring  activity  that  the  initial  costs  (i.e.,  "P")  will  be  excessive.  Applying  the 
life  cycle  cost  analysis  to  the  ATE  program  as  an  entity  will  result  in  a more 
constructive,  viable  procurement  posture.  As  indicated  in  the  Marcy  Report,  the  ATE 
becomes  under  utilized  and  ineffective  due  to  poor,  limited  or  nonexistent 
documentation. 
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Estdblishing  the  maintenance  philosophy  of  the  ATE,  or  modifying  the  mainte- 
nance philosophy  of  existing  ATE,  is  of  paramount  importance  during  the  ATE 
selection  process.  Providing  auto  check,  automatic  fault  detection  and  BIT  will  j 

minimize  the  maintenance  manhours  necessary  to  support  the  ATE.  ! 

The  documentation  requirements  for  the  ATE  must,  of  necessity,  address  a much 
broader  spectrum  of  performance  variables  than  the  UUT.  The  documentation  must  ! 

allow  for  resolution  of  conflicts  between  the  UUT  and  the  ATE,  as  well  as  isolation  to  | 

major  elements  within  the  ATE  system.  Further  isolation  to  the  faulty  major  element 
subassembly  and,  on  an  optimum  basis  to  the  piece  part.  Maintenance  of  the  ATE, 
through  constructive  use  of  a fully  comprehensive  design  guide  will  allow  the  ATE  to 
be  maintained  on  the  lowest  organizational  levels,  thereby  assuring  that  ATE  achieves 
its  primary  function  of  supporting  the  UUT. 

5. 3. 3. 3 Recommendations.  For  newly  developed  equipments,  the  maintenance 
philosophy  of  the  ATE  must  be  established  early.  These  requirements  should  reflect 
the  highest  possible  level  of  self  maintenance. 

The  documentation  phase  of  the  development  program  must  clearly  define,  by 
contract,  the  data  items  necessary  for  Fleet  support,  as  well  as  a technical  narrative 
to  describe  the  performance  requirements. 

Definition  of  an  overall  system  interface  specification,  as  well  as  a detailed 
document  package  to  verify  tlie  functional  interface  parameters,  is  mandatory.  Major 
element  performance  parameters  will  also  be  required  to  achieve  further  isolation.  It 
would  be  preferable  to  have  this  function  self-contained  within  the  ATE  system. 

In  order  to  resolve  the  problems  as  outlined  in  the  Marcy  Report  which  concern 
ATE  documentation  and  ATE  maintainability,  the  following  is  recommended: 

a.  Establish  a policy  which  will  require  the  highest  levels  of  ATE  self 
maintenance  (i.e.,  self  check,  self  test,  self  calibration,  etc.)  as  practical 
within  the  constraints  of  the  overall  program  performances  (i.e., 
cost/schedule). 

b.  This  policy  shall  define,  after  establishing  the  level  of  self  maintenance, 
the  minimum  data  required  for  support  of  the  ATE.  These  data  shall 
include,  but  should  not  be  limited  to: 

Operator/operation  instruction 
Overall  system  manual 
Major  functional  element  manual 
Diagnostic  manual 

Calibration  manual  (adjustment/alignment) 
will  be  a function  of  the  self  calibration  available 
Laboratory  calibration  manual. 
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c.  Establish  within  this  policy,  the  minimum  quality  acceptable  for  deliverable 
data  either  by  specifying  the  requirement  or  modifying  the  existing 
specifications  (e.g.,  MlL-D-1000)  as  noted  in  5. 3. 2. 3. 

d.  Establish  within  this  policy,  that  all  required  technical  support  documenta- 
tion be  delivered  concurrent  with  the  ATE  system  deployment. 

5.3.4  TPS  documentation. 

5.3.4. 1 Statement  of  the  problem.  TPS  development  programs,  most  often  produced 
by  a prime  weapon  systems  contractor,  have  not  provided  adequate  documentation  to 
accommodate  organic  support  utilizing  ATE.  The  result  has  been  increasing  logistics 
problems  in  terms  of  good  UUTs  being  returned  for  repair  and  bad  UUTs  being 
certified  RFI.  The  impacts  of  these  conditions  are  obvious  both  in  the  Fleet  readiness 
posture  as  well  as  O&MN  cost  impacts. 

The  deficiencies  concerning  TPS  documentation  (TPD)  have  been  stressed  in  the 
Marcy  Report.  The  Marcy  Report  details  the  interrelationship  of  the  TPS  data  to  the 
entire  ATE  system,  and  furthermore  stresses  the  impact  on  the  total  readiness  posture 
of  the  Fleet. 

5. 3. 4. 2 Analysis.  The  total  system  concept  must  again  be  applied  in  terms  of  the  TPS 
procurement.  The  hardware  procurement  cycle  must  encompass  the  TPS  development 
process.  Data  items  which  define  and  assist  the  organizational  level  test,  in  terms  of 
the  test  flow,  the  summary  instructions,  the  program  routine,  as  well  as  narratives 
which  provide  for  anomalies  caused  by  environmental  considerations  are  necessary.  In 
addition,  sufficient  documentation,  such  as  wiring  diagrams  and  active  element 
schematics  for  cabling  and  interconnection  devices  are  necessary. 

5. 3. 4. 3 Recommendations.  Structuring  the  TPS  data  and  documentation  requirements, 
in  consonance  with  a total  procurement  policy,  will  provide  for  definition  of  those 
items  necessary  to  achieve  supportability  of  the  TPS.  The  TPS  in  its  development  will 
be  a marriage  of  the  UUT  requirements  and  the  ATE  capability,  with  the  attendant 
documentation  providing  a road  map  of  test  conduct. 

The  documentation  must  provide  the  detailed  information  necessary  to  allow  a 
traceable  path  of  the  overall  test  philosophy,  the  testing  paths  utilized,  program  aids 
as  well  as  the  expected  results,  with  exceptions  and  deviations  to  the  expected  norms. 

A rigorous  prove-out  of  the  TPS  documentation  is  mandatory  to  ensure  that  the 
TPS  and  its  attendant  documentation  forms  a cohesive  package. 

The  Marcy  Report  established  a number  of  problem  areas  associated  with  TPSs. 
The  test  program  documentation  portion  of  the  total  problem  area  may  be  improved  by 
establishing  the  following  policy; 

a.  Require,  via  policy  statements,  that  project  managers  for  TPS  be 
thoroughly  conversant  with  the  documentation  requirements  of  AR-9B. 
This  must  include  suppplementary  data. 
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b.  Establish  that  TPD  be  developed  in  parallel  with  and  as  part  of  the  overall 
TPS  program. 

c.  Establish  a policy  which  requires  that  a rigorous  prove-out  of  the  TPD  be 
accomplished  during  the  TPS  verification  utilizing  non-engineering 
personnel. 

d.  Within  these  policies,  require  that  the  minimum  quality  standards  recom- 
mended in  5.3.2. 3 for  drawing  and  data  preparation  be  imposed  for  the 
TPD. 

e.  Establish  a policy  which  directs  that  the  TPD  delivery  be  concurrent  with 
delivery  of  the  TPS. 

5.4  Integrated  logistics  support  impact  on  automatic  test  equipment. 

5.4.1  Background.  Sparing,  training,  and  skill  level  requirements  have  been 
determined  in  the  past  by  a combination  of  methods  that  include  contractor 
recommendation,  and  review  of  3M  data  and  unsatisfactory  reports  by  specialist 
organizations  in  the  Navy.  Trade-offs  conducted  by  ILS  personnel  included  parameters 
such  as  cost,  maintenance  concept,  BlT/ATE  relationships,  location  of  the  ATE  (IMA, 
depot  or  both),  repair/discard  philosophies,  and  the  sparing  engendered  by  these 
parameters.  Trade-offs  did  not  always  take  into  account  fault  isolation  ambiguity 
group  sizes,  and  the  ATE/BIT  complementary  capabilities.  Support  for  automatic 
testing  has  had  its  problems  in  part,  due  to  the  limitations  of  these  methods,  and  to  a 
large  measure,  funding  and  ship  space  constraints.  The  result  has  been  increased 
turnaround  time  for  repairables. 

Because  of  these  complex  relationships,  some  tool  is  required  to  allow  an 
objective  trade-off  to  be  conducted  between  all  these  parameters.  The  objective 
should  be  to  arrive  at  a well  justified  mix  between  ATE  and  BIT,  and  the  number  of 
spares  required  for  supporting  the  prime  weapon  system  and  the  automatic  test 
equipment. 

5.4.2  Analysis.  Several  Navy  and  DoD  instructions  were  reviewed  that  apply,  even 
indirectly,  to  the  ILS  impact  on  ATE.  The  primary  documents  reviewed  were  NAVMAT 
Instruction  4000. 20B  - Integrated  Logistics  Support  Planning  Policy;  NAVMAT  P4000  - 
Integrated  Logistics  Support  Implementation  Guide  for  DoD  Systems  and  Equipments; 
NAVMAT  Instruction  4000.29  - Basic  Principles  of  Logistics.  Other  instructions  more 
directly  related  to  automatic  test  equipment  were  reviewed,  such  as  NAVAIR 
Instruction  4000.2A  - Integrated  Logistics  Support  for  Aeronautical  Systems  and 
Equipment;  NAVAIR  Instruction  4000.14  - Navy  Prepared  Logistic  Support  Plans  for 
Aeronautical  Systems  and  Equipment;  NAVAIR  Instruction  4000.12  - Integrated 
Logistic  Support  Management  Organization  and  Responsibilities;  and  NAVAIR 
Instruction  4790.2  - Maintenance  and  Support  Policy  Concerning  Weapon  Avionic 
Systems  Supported  by  VAST.  The  following  standards  were  also  reviewed  for  the 
impact  of  logistics  on  automatic  test  equipment:  MIL-STD- 1390.1  A - Level  of  Repair 
Analysis;  MIL-STD- 1388  - Logistic  Support  Analysis;  AR-60  -Level  of  Repair  for 
Aeronautical  Material;  and  MIL-STD- 1345,  Data  MeaT  jrement  in  Support  of  Mainte- 
nance, Calibration  and  Repair  of  Electronic  Equipment  . 
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The  Navy  has  referenced  automatic  test  equipment  in  their  logistics  documents, 
, for  example,  NAVMAT  Instruction  4000.20B  page  39;  NAVAIR  Instruction  4790.2- 

Maintenance  and  Support  Policy  Concerning  Weapon  Avionic  Systems  Supported  by 
VAST;  and  NAVAIR  Instruction  4000.14  - Navy  Prepared  Logistic  Support  Plans  for 
Aeronautical  Systems  and  Equipment,  page  7.  For  the  most  part,  the  logistics  support 
documents  listed  above  are  concerned  with  general  purpose  electronic  test  equipment, 
and  the  impact  on  logistics  analysis.  Very  few  computer  programs  remain  which  can 
trade  off  parameters  for  selecting  automatic  test  equipment.  As  described  in 
NAVMAT  Instruction  P4000,  there  are  several  typical  logistics  support  models  that  can 
trade  off  several  ILS  parameters.  For  instance,  the  OLF  program  which  trades  off 
total  logistics  cost  support  equipment  by  echelon  of  maintenence,  various  logistics 
times,  etc.,  the  LORAN  evaluates  alternative  support  postures  and  allows  for  trade-off 
evaluation  of  any  two  or  more  ILS  parameters  including  cost  support  equipment  by 
echelon  of  maintenance,  NOR  rates,  various  logistics  times,  etc.  There  are  also 
several  complex  operational  analysis  and  logistics  models  that  do  everything  from 
simulating  aircraft  operational  events  to  taking  into  account  resource  shortages,  flying 
schedules,  alert  commitments,  abort  rates,  attrition,  and  related  changes  in  concepts 
and  policies. 

The  instructions  cover  a wide  gamut  of  ILS  functions.  Actually,  ILS  functions 
are  impacted  by  a number  of  other  functions  which,  in  themselves,  would  relate  to 
automatic  test  equipment.  For  instance,  reliability,  maintainability,  human  factors, 
safety,  standardization,  and  quality  assurance  engineering.  All  of  these  disciplines 
impact  on  ILS  as  well  as  automatic  test  equipment.  Other  instructions,  such  as 
NAVAIR  Instruction  4000.12,  call  for  assuring  that  adequate  funds  are  budgeted  and 
made  available  for  development  of  ILS  in  each  system  or  equipment,  and  require  that 
ILS  requirements  approved  by  the  logistics  manager  are  included  in  all  purchase 
requisitions  for  all  systems  and  equipment,  for  which  the  ILS  manager  has  cognizance. 
Despite  these  instructions,  programs  go  forward  with  inadequate  sparing  and 
maintenance  funding. 

Quite  recently,  the  Navy  developed  a test  equipment  effectiveness  model  which 
trades  off  various  ILS  parameters  including  the  selection  of  different  kinds  of 
automatic  test  equipment,  and  built-in-test.  The  TEEM  model  as  noted  herein  requires 
further  refinement  to  be  fully  operational. 

Despite  the  fact  that  procurement  specifications  and  project  managers  require 
ILS  budgets  involving  SRA  and  WRA  modules,  test  programs  and  maintenance  aids,  the 
funds  allocated  for  these  items  are  often  deleted  or  reduced  due  to  economics  made  in 
programs.  When  a system  is  delivered  to  the  Fleet,  inadequate  numbers  of  spares  and 
test  program  sets  go  along  with  the  systems.  The  problem  still  exists,  even  though  the 
systems  are  bound  to  perform  integrated  logistics  support  analysis  of  various  kinds. 
Funding  constraints  cause  systems  to  be  fielded  that  are  not  supportable.  A case  in 
point  is  the  S-3A  aircraft  which  was  delivered  to  the  Fleet  with  a low  percentage  of 
the  SRA  test  programs  developed,  and  resulted  in  incomplete  diagnostic  and  fault 
isolation  capabilities  for  all  avionics  on  the  S-3A  supported  by  the  VAST  and  HATS 
systems.  Despite  the  fact  that  the  cognizant  test  program  maintenance  facility,  the 
Naval  Station,  Alameda,  had  requested  funds  and  manpower  to  develop  the  remaining 
test  program  sets  for  the  S-3A,  the  system  still  lacks  a complete  complement  of  test 
programs. 
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5.4,3  Recommendations.  As  a result  of  the  TEEM  model  partially  filling  a void  in  the 
trade-off  of  ATE  augmented  by  BIT,  and  the  logistics  support  required  for  the  ATE  and 
the  UUTs  that  the  program  indicates,  a policy  should  be  written  around  an  improved 
TEEM  model  invoking  its  use  on  all  ILS  programs  that  have  to  do  with  automatic  test 
equipment.  This  tool  should  also  be  referenced  in  the  acquisition  guide  for  automatic 
test  equipment.  For  the  short  term,  a NAVMAT  Instruction  governing  maintenance 
and  support  policy  concerning  automatic  test  equipment  should  be  written  around 
NAVAIR  Instruction  4790.2. 

The  ILS  direction  currently  promulgated  possesses  the  required  level  of  analysis 
and  control  to  assure  the  desired  program  success.  The  direction  and  timing  of  the  ILS 
processes,  however,  require  refinements,  re-emphasis,  and  cross-checks  to  form  a 
cohesive  program.  The  following  recommendations  are  offered  to  provide  these 
refinements: 

a.  Establish  and  promulgate,  as  guidance,  an  ILS  checklist  which  emphasizes 
and  provides  the  interrelationship  between  the  prime  equipment  ILS  and  the 
support  equipment  requirement.  The  checklist  should  address  specifically, 
the  support  considerations  which  must  be  established,  implemented  or 
reported  on  prior  to  submission  for  DSARC. 

b.  Establish  a policy  which  will  allow  generation  of  a companion  ILS  for 
support  equipments.  This  companion  ILS  must  have  funding  derived  from, 
but  independent  of,  the  prime  system  procurement  budget.  Dollar  limits 
will  have  to  be  established  which  will  allow  a determination  to  be  made  on 
initiation  of  companion  ILS  activity. 

c.  Establish  a policy  requiring  the  prime  system  TEMP  to  fully  address  the 
support  equipment  selection/procurement  process  for  both  hardware  and 
software  elements. 

d.  Establish  policy  which  requires  that  the  prime  system  ILS  and  companion 
support  system  ILS  be  integrated,  analyzed  and  updated  prior  to  submittal 
for  DSARC.  This  policy  should  also  direct  and  establish  that  the  companion 
ILS  be  initiated  immediately  after  DSARC  I (i.e.,  during  validation  phases). 

5,5  Contractual  requirements  for  automatic  test  equipment. 

5.5.1  Background.  Efforts  to  procure  TPSs  by  the  Navy  for  specific  avionics  or 
equipment  would  be  considerably  eased  with  the  development  of  standard  procurement 
specifications  for  SRAs  and  WRAs.  These  will  contain  requirements  to  verify,  validate 
and  control  configuration  of  TPSs.  These  documents  will  also  reference  the  standard 
TRD  now  under  development  by  the  Navy  which  may  be  in  the  form  of  a revised  MIL- 
STD-1519  or  AR-8,  9 and  10  documents. 
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The  S-3A  avionics  test  program  problems  originated  when  VAST  was  directed  as 
the  primary  support  equipment.  Seventy-five  percent  of  the  avionics  for  the  S-3A  had 
already  been  delivered.  The  resultant  lack  of  a consistent  maintenance  philosophy  on 
the  avionic  black  boxes  severely  complicated  the  fault  isolation  capability  of  VAST. 
Subcontracts  let  for  S-3A  SRAs  led  subcontractors  to  believe  that  they  would  supply 
the  support  equipment  for  their  SRAs.  The  SRA  designs  also  were  based  on  throw- 
away concepts  rather  than  fault  isolation. 

The  B-1  program  spent  many  man-hours  resulting  in  a multi- volume  procurement 
specification  for  their  functional  test  stations.  Within  this  specification  was  a 
specification  for  the  B-1  avionics  TPSs  based  on  a common  test  requirements 
document  for  all  TPSs.  Standard  data  item  descriptions  have  been  developed  for  the 
B-1  program  to  provide  direction  to  system  designers.  The  standard  DID  and  CDRL 
samples  are  enclosed  as  Appendix  F. 

5.5.2  Analysis.  Standard  DIDs  have  been  available  for  inclusion  in  procurement 
packages  that  relate  to  ATE/BIT.  DIDs  developed  for  the  B-1  and  S-3  programs  have 
been  analyzed  but  most  are  general  in  nature  and  do  not  solve  problems  in  validation, 
verification,  configuration  control,  standardization,  testability  and  other  important 
testing  problems.  There  were  DIDs  which  did  address  these  areas,  and  these  are 
meaningful  in  terms  of  overall  program  performance. 

By  specifying  certain  trade-off  study  results  as  deliverables,  the  DIDs  provide  an 
assurance  that  testability  and  ATE/BIT  considerations  are  integrated  into  the  design 
from  inception.  In  the  past,  these  studies  were  required  by  the  Navy  but  did  not 
require  that  the  data  be  delivered.  Contractors  were  only  obligated  to  show  results 
and  thus  did  not  include  detailed  analysis  which  explained  how  they  arrived  at  the 
results.  A contractor,  knowing  that  his  analysis  will  now  be  subject  to  careful  scrutiny 
and  approval,  will  be  more  diligent  in  developing  effective,  early-on  design  decisions 
with  regard  to  testability,  supportability  and  cost. 

5.5.3  Recommendations.  It  is  recommended  that  a NAVMAT  Instruction  containing 
standard  ATE-related  DIDs  be  issued.  It  is  also  recommended  that  a policy  be 
promulgated  requiring  DIDs  for  ATE  selection  and  instability,  etc.,  be  generated. 

5.6  Configuration  control  and  ATE. 

5.6.1  Background.  UUT  test  programs  must  be  developed  so  that  a minimum  number 
of  interface  devices  (IDs)  are  required.  However,  once  developed  and  accepted, 
changes  to  any  part  of  this  test  program  set,  which  includes  the  CUT,  ATE,  TRD  and 
test  program,  must  be  controlled  - that  is,  the  configuration  design  must  be  controlled. 
A chain  of  problems  occurs  when  the  ATE  manufacturer,  ID  designer,  white-hat  or 
avionics  manufacturer  makes  a modification  to  the  test  program,  or  design  of  the 
UUT,  and  none  of  these  engineering  activities  notifies  the  other.  What  results  is  a 
chain  of  developments  that  is  impossible  to  trouble-shoot.  The  problem  is  compounded 
by  the  use  of  contractors  to  develop  test  programs  or  make  engineering  changes 
without  regard  to  those  systems  already  In  the  Fleet,  and  without  regard  to  the  impact 
on  other  elements,  including  the  interface  devices.  For  example,  a contractor 
developing  a test  program  for  the  S-3  avionics  may  develop  a test  program  without 
testing  it  with  the  interface  device  available  with  the  HATS  system.  The  test  program 
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set  sent  to  the  aircraft  carrier  intended  to  use  the  system  has  a different  version  of 
the  HATS  and  a different  version  of  the  interface  device,  and  possibly  a different 
version  of  the  UUT  meant  to  be  tested.  Multiply  this  problem  by  the  number  of  prime 
weapons  systems  designed  to  be  supported  by  automatic  test  equipment,  and 
configuration  control  and  management  of  test  program  sets  becomes  critical. 

While  the  Navy  has  many  documents  that  address  themselves  to  configuration 
management  and  control,  few  of  these  documents  specifically  address  the  problem 
associated  with  test  program  sets.  The  section  that  follows  is  designed  to  analyze 
what  instructions  and  standards  exist  that  can  be  used  to  address  the  problem  of  TPS 
configuration  control  and  management. 

5.6.2  Analysis.  A handful  of  instructions  exist  that  address  the  problem  of 
configuration  management  of  hardware/software.  Among  these  are  the  NAVMAT 
Instruction  4 130.1  A - Department  of  Defense  Configuration  Management  Manual;  MIL- 
STD-480  - Configuration  Control  Engineering  Changes,  Deviations  and  Waivers;  MIL- 
STD-481  - Configuration  Control  Engineering  Changes  (Short  Form);  ^AVAIR 
Instruction  4130.1,  Changes  1,  5 and  8 - NAVAIRSYSCOM  Configuration  Management 
Manual;  NAVAIR  Instruction  4275. 3B  - Configuration  Control,  MIL-STD-480  and  481, 
Implementation  of;  AR-9B  - Test  Program  Sets,  General  Requirements  for;  and  MIL- 
STD-1519  - TRD,  Preparation  of. 

Currently,  AR-9B  and  MIL-STD-1519  are  being  rewritten  to  develop  a new  test 
requirements  document  military  standard.  Although  all  the  above  references  are  broad 
enough  to  cover  the  subject  at  hand,  that  is,  they  require  change  board  approval  of  any 
changes  to  hardware  or  software  or  the  interface  device,  none  of  these  are  specific 
enough  to  address  the  test  program  set  problem  as  a total  system.  A part  of  one 
instruction  approaches  the  test  program  set  configuration  control  problem:  NAVAIR 
Instruction  4130.1  Appendix  C (Configuration  Management  for  VAST).  However,  this 
instruction  is  written  around  the  VAST  test  stations.  It  does,  however,  correct  an 
earlier  deficiency  wherein  the  organizations  responsible  for  test  program  set 
development  are  made  voting  members  of  the  change  control  board.  The  instructions 
must  also  be  rewritten  to  a point  of  view  of  designating  specific  support  facilities  for 
other  automatic  test  equipments,  or  developing  the  general  criteria  for  designating 
these  facilities.  NAVAIR  Instruction  4275. 3B  requires  that  procurement  documents  for 
the  acquisition  of  NAVAIR  material  contain  a configuration  control  clause  invoking 
either  MIL-STD-480  or  481  depending  on  the  item  being  procured. 

Various  components  of  configuration  management  can  be  found  in  different 
instructions  and  standards.  For  example,  several  definitions  exist  for  Class  I and 
Class  II  engineering  changes.  MIL-STD-480  defines  a Class  I engineering  change  which 
contains  one  or  more  of  a number  of  factors.  One  of  the  factors  being  an  effect  on  the 
interface  characteristics  which  can  be  interpreted  as  changes  in  test  programs 
affecting  the  interface  device  between  the  UUT  and  the  ATE.  This  paragraph  is 
referenced  in  NAVMAT  Instruction  4130. lA.  AR-8B  VAST  Avionic  Systems 
Compatibility,  General  Requirements  for,  also  references  MIL-STD-480  in  defining 
Class  I engineering  changes.  Paragraph  4.1.3  of  AR-8B  defines  configuration  control 
requirements  of  TRDs.  In  general,  one  must  search  several  documents  to  arrive  at  an 
overall  configuration  management  procedure  for  ATE. 
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5.6.3  Recommendations.  It  is  recommended  that  a NAVMAT  Instruction  be  developed 
* which  is  written  around  the  combination  of  MlL-STD-480,  481,  and  NAVAIR  Instruction 


4130.1,  Appendix  C.  This  would  be  a short  term  recommendation,  and  minor  changes 
to  NAVAIR  Instruction  4120.1  Appendix  C would  be  the  basis  for  this  instruction.  For  j 

the  long  term,  an  addition  of  a section  to  MIL- STD-480  that  covers  test  program  set  J 

configuration  management  is  recommended. 
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Details  of  the  review  of  all  NAVMAT  instructions  are  contained  in  this  appendix. 
The  instructions  were  categorized  into  9 elements:  Acquisition  and  Planning;  ^ftware 
and  Configuration  Management;  Maintenance;  Logistics;  Personnel  and  Training; 
Finance;  Safety,  RM  and  QA;  ATE  and  Calibration*  and  Standardization.  Only  those 
instructions  directly  related  to  test  and  monitoring  systems  were  reviewed.  Each 
instruction  was  synopsized  and  recommendations  made  to  make  it  suitable  for  ATE 
considerations.  Although  there  were  no  voids  in  any  of  the  categories,  many  of  the 
instructions  were  unsuitable  for  revision  to  cover  certain  ATE  aspects.  As  a result  of 
this  review,  the  six  categories  listed  in  the  report  summary  were  chosen  as 
recommended  new  policies  and  procedures  to  fill  voids. 
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TABLE  A-8.  NAVMAT  INSTRUCTIONS 
ATE  AND  CALIBRATION 
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SECNAVINST  5000.1 
NAVMATINST  4000.20A 
NAVMATINST  4120.97A 
NAVMATINST  5300.9A 
MIL-STD  1309A 

COMMENT 

NO  REVISION  NECES- 
SARY 

NEEDS  UPDATE  TO 
SPECIFY  ATLAS  AS 
STANDARD  NAVY  TEST 
PROGRAMMING  LAN- 
GUAGE 

WHAT  HAS  BEEN  DONE 
WITH  RESULTS  OF 
INVENTORY? 

NEEDS  TQ  BE  EXPAND 
ED  WITH  LOGISTICS 
MODELS,  INVENTORY 
INFORMATION,  ETC. 
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TION OF 
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PLANNING) 

ATE  PROGRAMMING 
LANGUAGE,  INTERIM 
STD.  FOR 

INVENTORY  OF  AUTO- 
MATIC TEST  MONITOR- 
ING & DIAGNOSTIC 
SYSTEMS  & EQUIPMENT, 
REQUEST  FQR 

DATA  BANKS  FOR 
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MENT (TMDE)  ACTION 
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This  appendix  contains  a listing  of  ATE  related  DoD,  Air  Force,  Army,  OPNAV,  ] 

NAVAIR  and  NAVELEX  instructions.  The  technical  discussion  in  Section  5 of  this  ; 

report  revolves  around  many  of  these  instructions.  j 
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TABLE  B-1,  DOD  INSTRUCTIONS 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEGORY 

4100.3S 

OCT  1970 

DEVELOPMENT  OF  ILS  FOR  SYS- 
TEMS/EQUIPMENTS (l&L) 

LOGISTICS 

5000.29 

APR  1976 

MANAGEMENT  OF  COMPUTER  RE- 
SOURCES IN  MAJOR  DEFENSE  SYS- 
TEMS l8iL 

SOFTWARE  8(  CONFIGURA- 
TION MANAGEMENT 

5010.19 

JUL  1968 

CONFIGURATION  MANAGEMENT 
(DDRStE) 

SOFTWARE  & CONFIGURA- 
TION MANAGEMENT 

5010.21 

AUG  1968 

CONFIGURATION  MANAGEMENT 
IMPLEMENTATION  GUIDANCE 
(DDR81E) 

SOFTWARE  81  CONFIGURA- 
TION MANAGEMENT 

5126.43 

MAR  1970 

DOD  LOGISTICS  SYSTEMS  PLANNING 
(I8>L) 

LOGISTICS 

TABLE  B-2.  AIR  FORCE  & U.S.  ARMY 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEGORY 

AIR  FORCE 

LOGISTICS 

COMMAND 

VOL.  1 

SUPP.  NO.  1 
AFR  800-14 

15  OCT  1976 

MANAGEMENT  OF  COMPUTER 
RESOURCES  IN  SYSTEMS 

SOFTWARE  8i  CONFIGU- 
RATION MANAGEMENT 

AIR  FORCE 

LOGISTICS 

COMMAND 

VOL.  II 

SUPP.  NO.  2 
AFR  800-14 

18  OCT  1976 

ACQUISITION  8i  SUPPORT 
PROCEDURES  FOR  COMPUTER 
RESOURCES  IN  SYSTEMS 

SOFTWARE  8i  CONFIGU- 
RATION MANAGEMENT 

AIR  FORCE 

LOGISTICS 

COMMAND 

REG.  NO. 

66-37 

15  MAR  1976 

MANAGEMENT  OF  AUTOMATED 
TEST  SYSTEMS 

ATE  & CALIBRATION 

U.S.  ARMY 


TWENTY  COMMANDMENTS  OF 
SOFTWARE  ACQUISITION 


SOFTWARE  & CONFIGU- 
RATION MANAGEMENT 


TABLE  B-3.  OPNAVINST 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEGORY 

3960.11 

MAR  1976 

AUTOMATIC  TEST  MONITORING  8i 
DIAGNOSTIC  SYSTEMS  8i  NEW 
EQUIPMENTS 

ATE  & CALIBRATION 

4100.3A 

NOV  1972 

DEPARTMENT  OF  NAVY  ILS 

SYSTEM 

LOGISTICS 

4700.24  CH.  1 

JUN  1968 

FEB  1970 

POLICIES  GOVERNING  MAINTE- 
NANCE ENGINEERING  WITHIN  DOD 

MAINTENANCE 

4700.26 

JAN  1970 

MAINTENANCE  MANAGEMENT 
OBJECTIVES,  RESPONSIBILITIES 

FOR  REPORTING  OF 

MAINTENANCE 

4790.2A  CH.  1 
VOL.  I 


JUN  1973 


NAVAL  AVIATION  MAINTENANCE 
PROGRAM 


MAINTENANCE 


TABLE  B4.  NAVAIRINST 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEGORY 

1S00.2A 

NOV  1975 

TRAINING  FOR  MILITARY  PERSONNEL 
WITHIN  NAVAIR  REWORK  FACILITIES, 
POLICY  & PROCEDURE  FOR 

PERSONNEL  8i  TRAINING 

4000.2C 

JUN  1976 

ILS  FOR  AERONAUTICAL  SYSTEMS 
& EQUIPMENT 

LOGISTICS 

4000.10A 

JUN  1973 

REQUIREMENT  FOR  USE  OF  OOD 

GUIDE  4100JSG  OF  15  OCT  1968, 

ILS  PLANNING  GUIDE  FOR  DOD 
SYSTEMS  8i  EQUIPMENT 

LOGISTICS 

4000.12 

OCT  1973 
(REV.) 

ILS  MANAGEMENT  ORGANIZATION 

8i  RESPONSIBILITIES 

LOGISTICS 

4000.14A 

JUN  1975 

NAVY  PREPARED  ILSPS  & OLSPS  FOR 
AERONAUTICAL  SYSTEMS  8i  EQUIP- 
MENT 

LOGISTICS 

4130.1 

SEP  1970 

CONFIGURATION  MANAGEMENT 
MANUAL 

SOFTWARE  Si  CONFIGUR- 
ATION MANAGEMENT 

4275.3B 

OCT  1969 

CONFIGURATION  CONTROL 

SOFTWARE  Si  CONFIGUR- 
ATION MANAGEMENT 

4330.8A 

FEB  1976 

RESPONSIBILITIES  & PROCEDURES 

FOR  ACQUISITION,  USE  8i  DISPOSI- 
TION OF  SPECIAL  TOOLING  & STE 
ACCOUNTABLE  UNDER  NAVAIR  SUP- 
PLY 8i  SERVICES  CONTRACTS 

ACQUISITION  Si  PLANNING 

4411.1 

APR  1971 

PROCEDURES  FOR  IMPLEMENTATION 

OF  THE  COMMON  GSE  RETIREMENT 
PROGRAM 

ACQUISITION  Si  PLANNING 

4440.9A 

AUG  1975 

REQUEST  FOR  INVENTORY  OF  ATE 

ATE  Si  calibration 

4441.2 

DEC  1972 

INVENTORY/IN-USE  MANAGEMENT  OF 
AN/USM  247(V)  VAST  ASSOCIATED 

GSE  APPEARING  IN  THE  QV-1  Si 

QV-2  ALLOWANCE  LISTS 

ATE  Si  calibration 

4700.6A 

SEP  1971 

PLANNING,  PROGRAMMING,  BUDGET- 
ING, PROCUREMENT  8i  ILS  OF  CGSE 
END  ARTICLES;  PROCEDURES  8i 
RESPONSIBILITY  FOR 

LOGISTICS 

JAN  1976 

GSE  LOGISTICS  DIVISION 

LOGISTICS 

60 


TABLE  B-4.  NAVAIRINST  (Cont'd) 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEOGRY 

4700.17 

JUN  1974 

POLICIES  GOVERNING  MAINTENANCE 
ENGINEERING  WITHIN  NAVAIR- 
SYSCOM 

MAINTENANCE 

4720.4 

MAR  1976 

IN  SERVICE  ENGINEERING  PROGRAM 
FOR  ATE  TEST  PROGRAMS 

ATE  & CALIBRATION 

4790.2 

FEB  1973 

MAINTENANCE  & SUPPORT  POLICY 
CONCERNING  WEAPON/AVIONIC 
SYSTEMS  SUPPORTED  BY  VAST 
(AN/USM-247(V)) 

ATE  8i  CALIBRATION 

5400.18 

OCT  1969 
(REV.) 

ESTABLISHMENT  OF  PROGRAM 
MANAGER  FOR  GROUND  SUPPORT 
EQUIPMENT 

ACQUISITION  & PLANNING 

5400.67 

MAR  1972 

CERTIFICATION  PROGRAM  FOR  NAVY 
AIR-LAUNCHED  GUIDED  WEAPON 

TEST  SYSTEM 

ATE  & CALIBRATION 

5400.72 

JUN  1973 

POLICY  & RESPONSIBILITY  FOR  THE 
SELECTION,  DESIGN,  APPROVAL, 
ORDERING,  DELIVERY  & LOGISTICS 
SUPPORT  OF  GSE 

ACQUISITION  8i  PLANNING 

13800.4 

JUL  1968 

GSE  FOR  WEAPONS  SYSTEMS  UNDER 
AUGMENTED  SUPPORT;  PROCEDURES 

8i  RESPONSIBILITIES  FOR 

MAINTENANCE 

AR8B 

24  DEC 
1974 

VAST/AVIONICS  SYSTEM  COMPATI- 
BILITY, GENERAL  REQUIREMENTS 

FOR 

ATE  & CALIBRATION 

AR9B 

VAST  TEST  PROGRAMS,  GENERAL 
REQUIREMENTS  FOR 

ATE  & CALIBRATION 

AR10A 

16  JUN 
1969 

NOTICE  1 

MAINTAINABILITY  OF  AVIONICS 
EQUIPMENT  8i  SYSTEMS,  GENERAL 
REQUIREMENTS  FOR 

ATE  & CALIBRATION 

AR30A 

ILS  REQUIREMENTS  FOR  AS  8i  EQUIP- 
MENTS 

ATE  8(  CALIBRATION 

61 


TABLE  B-5.  NAVELEXINST 


INSTRUCTION 

NUMBER 

DATE 

TITLE 

CATEGORY 

4200.8B 

FEB  1972 

ACQUISITION  PROCEDURES  FOR 
PRODUCTION  HARDWARE 

ACQUISITION  & PLANNING 

63 


APPENDIX  C 


This  appendix  contains  ATE  related  military  specifications.  These  specifications  j 

are  illustrated  in  Figure  C-1.  This  appendix  also  provides,  in  matrix  format,  items  ; 

which  define  the  BlT/test  point/interface  bus  specifications,  and  provides  content  and  3 

commentary  for  each  document.  < 
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MIL  STO  4150 
TEST  PROVS.  FOR 
ELECTRON.  SYS 
ASSOC.  EQPMT 
DESIGN  CRIT. 
FOR  (ALL  000) 


MIL  T 21200L 
TEST  EQPMT  FOR 
USE  WITH  ELECT. 
AND  ELECTRONIC 
EQPMT.  (ALL  DODI 


MIL  T 24309 
TECH  SUPPORT 
PLAN  ELECTRONIC 
EQUIPMENT 


I MIL-STD  1533 
I DIGITAL  DATA  INTER-  I 
I FACES  TO  SHIPBOARD  I 
I MULTIPLEX  BUS  | 

I SYSTEMS  (SMSI  j 


MIL-STD  1326 
TEST  PTS,  TEST  PT 
SELECT.  & INTER- 
FACE RQMTS  FOR 
EQPMT  MONIT.  BY 
SHIPBO  QN-LINE  ATE 


^ MIL-STD  454D  \ 

31  AUG  1973 
STD.  GEN.  REQTS  FOR 
ELECTRONIC  EQPMT 
APPROVED  ALL  OOO 
RM.  HF  & TEST  PROV. 
RQMT  32  SAFETY 
RQT  54 


MIL  T 2S800A 
TEST  EQPMT  FOR 
USE  WITH  ELECT. 
& ELECTR.  EQPMT. 
GEN.  SPEC.  FOR 
(ALL  OOD) 


MIL-STD  1309B 
DEFINITIONS  OF 
TERMS  FOR  TMDE 


MIL  E 8189G 
ELECTR.  EQPMT 
MISSILES,  BOOSTERS 
8i  ALLIED  VEHICLES 
GEN.  SPEC.  FOR 
(ALL  0001 


MIL  S 85120 
SUPPORT  EQUIPMT 
AERONAUTICAL, 
SPEC.  GEN.  SPEC. 
FOR  DESIGN  OF 
(ALL  DOD) 


MIL-STD  1553 
AIRCRAFT  INTERNAL 
COMMAND/RESPONSE 
TIME  DIV.  MULTI- 
PLEX DATA  BUS 


MIL-STD  883 
TEST  METHODS  8i 
PROCEDURES  FOR 
MICRO  ELECTRONICS 


MIL-STD  471 
MAINTAINABILITY 
DEMONSTRATION 


MIL  L 22769A 
LAUNCHER,  WEAPONS 
AIRBORNE  & ASSOC. 
EQPMT.,  GEN.  SPEC. 
FOR 


MIL-STD  488 
IEEE  STD  DIGITAL 
INTERFACE  FOR 
PROGRAMMABLE 
INSTRUMENTATION 


MIL-STD  1369 
ILS  PROGRAM 


MIL-STD  1519 
TRD 

PREPARATION  OF 


MIL-STD  1390 
LEVEL  OF  REPAIR 
ANALYSIS 


MIL-STD  1399 
INTERFACE  STAN- 
DARD FOR  SHIPBO 
SYSTEMS 


MIL-STD  1387 
PROCEDURES  FOR 
ACQUISITION  OF 
NON-STD  GPETE 


MIL-STD  1513 
TRADE  STUDIES 
FOR  THE  SELECT. 
OF  AVIONIC  TEST 
SUPP.  SYS;  CRITERIA 
FOR 


MIL-STD  480,481 
CONFIG.  CONTROL 
ENGIN.  CHANGES 
DEVIATIONS  AND 
WAIVERS 


TABLE  C-1.  ATE-RELATED  MILITARY  STANDARDS 


MIL-S-8512D 

SUPPORT  EQUIPMENT,  AERONAUTICAL,  SPECIAL.  GEN.  SPEC  FOR  DESIGN  OF 

MIL-T-21200L 

TEST  EQUIPMENT  FOR  USE  WITH  ELECTRICAL  & ELECTRONIC  EQUIPMENT, 
GEN.  SPEC.  FOR 

MIL-T-21578A 

TEST  EQUIPMENT,  HAZARDOUS  LOCATION,  INSTALLED,  BASIC  RQMTS  FOR 

MIL-T-24309 

(SHIPS) 

TECH.  SUPPORT  PLAN  - ELECTRONIC  EQUIPMENT 

MIL-T-288(X>A 

TEST  EQUIPMENT  FOR  USE  WITH  ELECTRICAL  AND  ELECTRONIC  EQUIP- 
MENT. GEN.  SPEC.  FOR 

MIL-W-7622B 

WEAPON  SYSTEMS,  GUIDED  MISSILES,  GEN.  SPEC.  FOR 

MIL-STD-1309B 

DEFINITIONS  OF  TERMS  FOR  TEST,  MEASUREMENT  & DIAGNOSTIC  EQUIP- 
MENT 

IVIIL-STD-287 

TEST  SIGNALS  FOR  ELECTRONIC  TACTICAL  AIR  NAVIGATION  EQUIPMENT 

MIL-M-9901A 

(USAF) 

TECHNICAL  MANUAL,  TAPES,  CARDS 

MIL-STD-480 

CONFIGURATION  CONTROL  - ENGINEERING  CHANGES,  DEVIATIONS  AND 
WAIVERS 

MIL-STD-482 

CONFIGURATION  STATUS  ACCOUNTING  DATA  ELEMENTS  AND  RELATED 
FEATURES 

MIL-P-15137C 

(SHIPS) 

PROVISIONING  TECHNICAL  DOCUMENTATION 

MIL-HDBK-472 

MAINTAINABILITY  PREDICTION 

Ml  L-P-24052 

TEST  PROCEDURES  RADIAC  INSTRUMENTS 

MIL-S-24277 

(SHIPS) 

STANDARDIZATION  PROGRAM 

AD-837694 

A GUIDE  TO  THE  APPLICATION  OF  BIT  TO  NAVY  AVIONIC  EQUIPMENT 
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TABLE  C 2.  MAJOR  BIT/TEST  POINT/INTERFACE  BUS  RELATED  DOCUMENT  SUMMARY  (Cont'd) 
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kL.  A 


APPENDIX  E 


EXCERPTS  FROM 
REPORT  OF  INDUSTRY 

AD  HOC  AUTOMATIC  TEST  EQUIPMENT  PROJECT 
VOLUME  III  MANAGEMENT  CONSIDERATIONS 


IV  - TASK  F - OPERATIONAL  READINESS  MONITORING  SYSTEM  (ORMS) 


The  NELC  work  to  date  has  clearly  established  techniques  for  utilizing  and 
displaying  ORMS  related  data  of  the  ship  subsystems.  However,  the  one  thing  that  was 
not  clear  was  what  criteria  should  be  used  in  a subsystem  design  for  determining  the 
subsystem  output  data  requirements  for  ORMS.  In  general,  the  test  points  previously 
designed  into  the  subsystems  were  intended  for  use  in  performance  monitoring  and 
maintenance.  It  has  to  be  determined  how  to  use  the  existing  test  points  and  outputs 
to  establish  the  various  levels  of  Operational  Readiness.  Each  subsystem  requires  an 
analysis  to  determine  its  operational  readiness  levels.  It  is  apparent  that  a military 
standard  or  specification  is  required  that  defines  the  guidelines  to  be  used  in  specifying 
ORMS  test  points  or  data  sources.  A modified  MlL-STD-1326  was  considered  to  be  a 
good  vehicle  for  specifying  ORMS  related  test  point  requirements. 

The  task  group  has  prepared  an  ORMS  Implementation  Plan  as  an  addendum  to 
this  report  which  outlines: 

o The  evaluation  of  the  ORMS  concept  on  an  existing  ship  class  (FF-1052). 

o The  evaluation  of  the  ORMS  concept  for  a future  ship  class  (FF-7  LBTC). 

o The  preparation  of  a modified  MlL-STD-1326  for  use  as  monitoring, 

interface  and  test  point  standard. 
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TASK  F ADDENDUM 


OPERATIONAL  READINESS  MONITORING  SYSTEM  (ORMS) 
IMPLEMENTATION  PLAN 


9.  MODIFICATION  OF  MIL-STD-1326 

The  scope  of  MIL-STD-1326  is  to  describe  the  requirements  for  providing  test 
points  and  test  interfaces  in  shipboard  prime  equipment;  to  provide  criteria  for 
guidance  in  optimum  test  point  selection;  and  to  define  test  point  interface 
requirements,  data  generation  requirements,  and  test  point  selection  documentation 
procedures. 

Some  of  the  areas  in  which  MIL-STD-1326  should  be  revised  to  reflect  ORMS 
requirements  are: 

o The  standard  should  include  an  overall  shipboard  system  testing/ORMS 
philosophy  that  will  result  in  the  proper  degree  of  BITE  in  new  developirrent 
hardware. 

o The  standard  should  define  the  interface  requirements  between  internal 
BITE  and  external  functions  such  as  GPETE,  ORMS,  etc. 

o The  standard  should  define  which  standard  interfaces  are  mandatory;  i.e.,  if 
the  designer  chooses  to  bring  out  analog  or  digital  signals,  what  is  the 
analog/digital  interface? 

o Overall  system  guidance  regarding  the  total  ship  ORMS  must  be  provided 
with  respect  to  logical  processing  required  of  the  designer's  equipment 
relative  to  ORMS,  availability  of  external  stimuli,  and  external  multiplex- 
ing control. 

o Guidance  is  required  regarding  the  difference  between  test  points  that  are 
brought  to  locally  accessible  points  (such  as  card  edge)  and  test  points  that 
are  brought  to  a rear-panel  connector  for  distribution  (possibly  via  a 
multiplex  bus  system)  to  remote  monitoring/processing  in  other  equipment 
or  systems. 

o Clarification  of  the  definition  of  passive  sensor  is  required  in  light  of  the 
desire  to  encourage  standard  digital  interfaces. 
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VI  - TASK  I - SPECIFICATION  REVIEW 


MIL-STD-I326  and  the  draft  MIL-STD-I326A  are  concerned  with  test  points,  test 
point  selection  and  interface  requirements  for  equipment  monitored  by  shipboard  on- 
line ATE.  These  standards  are  of  major  concern  to  the  Task  F effort  on  Operational 
Readiness  Monitoring  Systems  (ORMS).  Pertinent  recommendations  from  the  Task  F 
report  are  presented  with  the  comments  of  the  Task  I Specification  Review  group. 
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APPENDIX  F 


RECOMMENDED  NAVMAT  INSTRUCTIONS 


APPENDIX  F 


This  appendix  provides  sample  NAVMAT  instructions  to  establish  policy  for 
(1)  Testability  of  New  Systems,  and  (2)  Standard  ATE  Related  Data  Item  Descriptions 
(DID).  Sample  DIDs  are  provided  herein. 

The  NAVMATINST  numbers  assigned  are  arbitrary  and  do  not  reflect  actual 
instructions  generated. 
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DEPARTMENT  OF  THE  NAVY 
Headquarters  Naval  Material  Command 
Washington,  D.C.  20360 


NAVMATINST  3960.10 
MAT  04T/GWN 
1 September  1977 


NAVMAT  INSTRUCTION  3960.10 


From:  Chief  of  Naval  Material 


Subj:  Testability  of  New  Systems 


Ref: 

(a) 

NAVMATINST  3910. 5C  subj:  Technical  Development  Plans  (TDPs) 
and  Research  and  Development  Planning  Summaries  (DD  Forms  1634); 
Procedures  for  Preparation,  Submission  and  Distribution 

(b) 

NAVMATINST  3960.9  subj:  Acquisition  Planning  Guide  for  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment;  Built-in- 
Test  (BIT)  Design  Guide;  Promulgation  of 

(c) 

NAVMATINST  4000. 20B  subj:  Integrated  Logistics  Support  (ILS) 
Planning  Policy 

(d) 

NAVMATINST  3960. 4A  subj:  Policy  and  Responsibility  for  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment 

1.  Purpose.  To  assist  project  managers  in  their  efforts  to  encourage  contractors  to 
design  a supportable  system  at  a reduced  life  cycle  cost. 


2.  Background.  The  design  of  a military  system  and  its  elements  has  in  the  past 
often  been  driven  only  by  performance  considerations,  while  maintenance  was 
considered  as  an  afterthought.  Design  of  special  test  equipment  to  test  the  system 
was  accomplished  after  the  fact  as  best  as  could  be  expected.  By  utilizing  a design 
team  composed  of  representatives  from  design,  test,  maintainability,  logistics, 
reliability,  systems  analysis,  packaging,  and  software/firmware,  many  of  the  down- 
stream maintenance  and  logistics  problems  can  be  avoided.  Reference  (a)  provides  a 
checklist  for  each  of  the  disciplines  during  the  development  phase  of  a program. 


Specifying  in  the  RFP  or  systems  specification  that  each  element  within  the 
system  must  be  designed  for  testability  results  in  lower  maintenance  and  life  cycle 
costs,  higher  operational  readiness  and  savings  in  space  for  and  the  need  of  special  test 
equipment. 
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NAVMATINST  3960.10 
1 September  1977 


Current  technology  enables  the  design  of  built-in-test  with  a very  low  overhead 
growth  (Ref.  (b))  provided  the  designer  makes  each  design  decision  with  other  team 
members.  Development  of  the  maintenance  concept  in  the  past  occurred  after  the 
design  was  developed  or  without  knowledge  of  the  detailed  design.  The  designer,  on 
the  other  hand,  was  not  required  to  consider  the  mission  or  the  maintenance  policy. 

3.  Policy.  Project  and  acquisition  managers  shall  require  the  establishment  of  a 
contractor  design  team  with  representatives  from  design,  test,  logistics,  maintainabil- 
ity, systems  analysis,  reliability,  software/firmware,  and  packaging  as  members.  From 
inception  of  the  design  each  design  decision  shall  be  approved  by  all  members  of  the 
team.  Frequent  design  reviews  held  with  appropriate  members  of  the  team  and  others, 
chaired  by  the  project  manager  or  his  representative,  shall  be  conducted.  As  a 
minimum,  decisions  such  as  placement  of  test  points,  accessibility,  partitioning,  bus 
arrangements,  redundancy,  fault  detection  and  isolation  shall  be  made  by  the  design 
team.  Sufficient  notice  and  information  on  the  latest  design  configuration  shall  be 
given  all  review  members  prior  to  the  review.  Trade  studies  concerning  BIT  level,  BIT 
versus  ATE,  optimum  test  point  selection,  software  versus  hardware  BIT,  repair- 
discard  level  and  their  relationship  to  mean  time  to  repair  and  mean  time  between 
failure  shall  be  conducted  during  this  design  process.  Methodology  for  developing  a 
figure  of  merit  for  the  systems'  testability  shall  be  developed  by  each  contractor 
subject  to  the  approval  of  the  project  office.  Design  improvements  during  the  design 
process  shall  be  measured  against  this  baseline. 

Scope  and  Applicability.  This  policy  applies  to  all  Systems  Commands,  CNM- 
designated  Project  Managers'  Offices  and  CNM  laboratories,  and  includes  all  platform, 
system  and  equipment  acquisitions  for  which  ATE/BIT  applications  are  established  or 
projected.  This  includes  self-test,  and  all  automatic  and  semi-automatic  test  and 
monitoring  equipment  as  defined  in  Reference  (d). 

5.  Action. 

a.  Addressees  shall: 

(1)  Implement  the  policy  as  stated  through  specific  reference  to  this 
document  in  applicable  acquisition  planning  and  procurement  direc- 
tives and  review  procedures. 

(2)  Ensure  that  new  and  planned  acquisition  programs  are  structured  and 
implemented  to  reflect  the  guidelines  provided. 

b.  The  Naval  Material  Command  (MAT  04)  shall  monitor  the  overall  adequacy 
of  ATE  acquisition  planning  and  implementation  on  the  basis  of  conformance  with  this 
policy. 
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DEPARTMENT  OF  THE  NAVY 
Headquarters  Naval  Material  Command 
Washington,  D.C.  20360 

NAVMATINST  3960.11 
MAT  04T/GWN 
1 September  1977 


NAVMAT  INSTRUCTION  3960.11 
From:  Chief  of  Naval  Material 

Subj:  Standard  ATE  Related  Data  Item  Descriptions  (DlDs) 

Ref:  (a)  NAVMATINST  3960.9  subj:  Acquisition  Planning  Guide  for  Automatic 

Test,  Monitoring  and  Diagnostic  Systems  and  Equipment;  Built-in- 
Test  (BIT)  Design  Guide;  Promulgation  of 

(b)  NAVMATINST  4000. 20B  subj:  Integrated  Logistics  Support  (ILS) 
Planning  Policy 

(c)  NAVMATINST  3960.4A  subj:  Policy  and  Responsibility  for  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment 

Enel:  Data  Item  Descriptions  and  Contract  Data  Requirements  Lists 

1.  Purpose.  To  assist  project  managers  to  specify  data  required  for  ATE/BIT 
acquisitions.  Application  of  the  enclosure  in  the  acquisition  process  of  platforms, 
systems  and  equipment  (end  items)  will  provide  assurance  of  proper  deliverables  for 
effective  maintenance  of  the  end  items. 

2.  Background.  Development  of  end-items  requires  that  support  equipment  be 
developed  concurrently.  With  the  advent  of  complex  end-items,  the  need  to  develop 
automatic  test  equipment  (ATE)  to  detect  and  isolate  faults  for  maintenance  purposes 
and  built-in-test  (BIT)  for  readiness  indications  became  urgent.  Along  with  the 
development  of  ATE/BIT,  the  requirements  for  test  programs,  operator  manuals,  test 
specifications,  etc.,  was  generated.  To  properly  specify  how  these  deliverables  are  to 
be  developed,  so  that  the  Fleet  will  understand  the  operation  of  the  hardware  and 
software,  standard  DIDs  and  CDRLs  have  been  developed  (enclosure). 

3.  Policy.  Enclosures  are  provided  for  use  by  project  managers  and  other 
acquisition  managers  to  acquire  ATE  and  BIT  for  their  end-items.  The  enclosures 
cover  acceptance  and  qualification  testing  data,  unit  under  test  (UUT)  test  plans  and 
procedures,  test  reports,  users  manuals,  software  specifications  and  self-test  design 
analysis.  The  enclosures  are  presented  as  guidance  and  do  not  necessarily  represent  a 
complete  DID/CDRL  package,  although  for  most  ATE/BIT  they  will  suffice.  • 
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NAVMATINST  3960.1  1 
1 September  1977 


4.  Scope  and  Applicability.  The  foregoing  policy  applies  to  all  Systems  Commands, 
CNM-designated  project  managers'  offices  and  CNM  laboratories,  and  covers  all 
platform,  system  and  equipment  acquisitions  for  which  ATE  applications  are 
established  or  projected.  ATE  applications  include  BIT,  self-test,  and  all  automatic 
and  semi-automatic  test  and  monitoring  equipment  as  defined  in  Reference  (a). 

5.  Action. 

a.  Addressees  shall: 

(1)  Implement  the  policy  as  stated  through  specific  reference  to  this 
document  in  applicable  acquisition  planning  and  procurement  direc- 
tives and  review  procedures. 

(2)  Ensure  that  new  and  planned  acquisition  programs  are  structured  and 
implemented  to  reflect  the  guidelines  provided. 

b.  The  Naval  Material  Command  (MAT  04)  shall  monitor  the  overall  adequacy 
of  ATE  acquisition  planning  and  implementation  on  the  basis  of  conformance  with  this 
policy. 
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NAVMAT  INSTRUCTION  3960.11 


L 


ENCLOSURE 


ATE/BIT  RELATED  DIDs/CDRLs 


lOCNTiriCATIOM  NOISI. 


DATA  ITEM  OESCXIPTIOH 


UUT  ACCEPTANCE  TEST  PROCEDURE 


1.  OCSCni«>TtOM/ AUMPOSC 


This  data  item  provides  procedures  for  tests  (and  inspec- 
tions) to  be  conducted  for  determining  acceptability  of 
Subsystem/Equipment  to  be  furnished  to  the  Navy. 


Awnov  AC  DATS 


1.  OVFieC  OP  PMIMAMV 
RUPONdMCITV 


This  information  is  used  to  equate  the  Contractor's 
acceptance  test  activity  to  the  acquisition  specification 
test  requirements. 


«.  ooc  RCQumco 


ARRROVAC  LIMIT  ATIOM 


A^^CICATICN/IMTCMRCI.  ATIOMSMI^ 


Applies  to  acquisitions  involving  new  or  modified  hardware 
end  items  where  acceptance  testing  is  required. 


<«C  Ca  (Mtndmterr  cilarf  in 


10.  PRCRARATION  IMarnuCTIOMS 


MIL- STD  1519 
AR  8,  9 & 10 


Mcac  MUMBcmai 


Acceptance  test  procedures  shall  be  prepared  as  working  documents  containing 
sufficient  detail  so  that  they  may  be  used  for  conducting  the  tests  to  which 
they  apply.  They  shall  include  the  following  information  as  a minimum,  and  may 
be  prepared  in  Contractor's  format. 

1.  Document  identification  number,  date,  and  revision  information. 

2.  Manufacturer's  name  and  division,  as  applicable. 

3.  Signature(s)  of  supplier  personnel  responsible  for  validity  of  the  document. 

4.  Description  of  test  item,  including  manufacturer's  part  number  and  name  of 
item. 

5.  Number,  date,  and  revision  of  applicable  procurement  specification.  If  desired, 
the  parenthetical  expression  "(latest  contractual  revision)"  may  be  used 
following  the  specification  number  in  lieu  of  date  and  revision  information. 

6.  Test  number  for  each  specific  test  described,  and  sequence  with  regard  to 
associated  tests. 


7.  Title  of  each  specific  test  described. 

8.  Correlation  of  each  test  to  the  corresponding  paragraph  number(s)  of  applicable 
acquisition  specification. 
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, R ARCS 


UUT  ACCEPTANCE  TEST  PROCEDURE 


Page  2 of  2 


Preparacion  Instructions  (Continued) 

9.  Quantity  to  be  tested  (specify  all  parts,  or  else  describe  the  intended 
sampling  plan) . 

10.  Quantitative  and/or  qualitative  success  criteria  for  each  test. 

11.  Action  to  be  taken  if  success  criteria  are  not  achieved. 

12.  The  environmental  conditions  under  which  the  test  is  to  be  conducted,  with 
tolerances . 

13.  The  test  configuration,  including  mounting  position/attitude  of  the  test  item 
during  the  test. 

14.  If  applicable,  the  power  supplied  to  the  item  under  test,  with  tolerances. 

15.  A detailed  statement  of  exactly  how  each  test  is  to  be  performed.  The  level 
of  detail  shall  be  such  that  the  statement  may  be  used  as  direct  and  adequate 
instructions  to  performing  test  personnel  experienced  in  the  type  of  testing 
described  but  not  necessarily  knowledgeable  of  the  item  to  be  tested. 

16.  Data  to  be  recorded  and  data  reduction  and  analysis  techniques.  Include 
sample  data  sheet(s). 

17.  Description  of  support  equipment  and/or  special  test  facilities  required. 

18.  Description  of  instrumentation,  including: 

a.  Sensing  and  recording  devices  (name,  range,  rated  accuracy  and,  if 
appropriate,  manufacturer  and  model  number.  For  rated  accuracy 
expressed  in  percent,  specify  percent  of  reading  or  percent  of  full- 
scale  indication,  as  applicable.) 

b.  Parameters  to  be  sensed  or  recorded,  correlated  to  the  devices  to  be  used. 

c.  Calibration  frequency,  methods  of  calibration,  and  calibration  standards 
(if  applicable,  acknowledge  compliance  with  MIL-C-45662A  without  pre- 
senting detailed  statement  of  standards  and  methods.) 

d.  Checkout  and/or  standardization  procedure  if  required  in  addition  to 
calibration . 

e.  A requirement  for  instrument  error  to  be  taken  into  account  in  determin- 
ing allowable  limits  of  instrument  readings. 

19.  Identification  of  figures  and  tables. 

20.  Definitions  of  uncommon  acronyms,  symbols,  and  abbreviations. 
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DATA  ITEM  OESCRIPTiCH 


*• 


lOCNririCATiOM  mooi 


USERS  MANUAL  (TEST  PROGRAMS) 


9.  OCSCIIIPTION/^UA^OSC 


A^^nov  AC  OAT  c 


This  manual  is  used  to  provide  the  Navy  and/or  contractor  ».  officc  or  aaimaat 
personnel  with  the  necessary  instructions  concerning  usage  ««srowai«iciTv 
of  the  test  program  and  instructions  on  how  it  is  to  be 
operated . 


«.  OOC  ACQUIRCO 


9.  A^riEOVAC  UlMlTATtOM 

APrciCATlON/INTKfIfECC  ATlOHtMl# 

The  basis  for  the  development  of  the  Users  Manual  is  the 

Navy's  specifications.  The  manual  will  normally  be  sub-  r'  m 

mitted  Co  the  procuring  activity  at  the  time  of  acceptance  woc* 
of  the  test  program  to  which  it  relates. 

AR-9B 


MCM.  NuMasms 


to.  *MS»AMATI0N  IMaTKUCTIONS 


Content . The  user-s  manual  for  a test  program  system  shall  contain  the 
following : 

a.  Introduction . A description  of  the  manual's  purpose,  scope,  organization, 
and  content . 

b.  Glossary.  Terms  used  in  the  manual. 

c . Test  Program  Capabilities: 

(1)  Purpose.  A description  of  the  purpose  of  the  test  program. 

(2)  General  Description.  A description  of  the  system,  giving  an  overview 
of  the  system  and  its  operation. 

(3)  Functions  Performed.  Identification  of  Che  specific  functions 
performed  by  the  test  program.  (This  may  be  in  terms  of  systems 
operations,  uses,  outputs,  etc.) 

d.  Function  Description.  A description  of  each  specific  function  within  the 
test  program.  The  following  subparagraphs  shall  be  repeated  for  each 
function: 

(1)  Title  of  function.  A descriptive  title  of  the  specific  function. 
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(2)  Description  of  function.  A summary  description  of  the  specific 
function,  including: 

(a)  Purpose  and  uses  of  function. 

(b)  Description  of  system  inputs. 

(c)  Description  of  expected  output  and  results. 

(d)  Relationship  to  other  functions. 

(e)  Summary  of  function  operation. 

e.  Usage  Instructions.  How  to  use  each  specific  function.  This  shall 
include  the  following : 

(1)  Preparation  of  inputs.  A definition  of  the  system  inputs,  other 

than  those  required  to  operate  the  test  program  (see  paragraph  lf(2)). 
These  inputs  constitute  the  basic  data  that  are  to  be  processed  by 
the  test  program.  The  definition  shall  include: 

(a)  Title  of  inputs. 

(b)  Description  of  inputs. 

(c)  Purpose  and  use. 

(d)  Input  media. 

(e)  Limitation  restrictions. 

(f)  Format  and  content. 

(g)  Sequencing  (e.g.,  formalized  deck  structure). 

(h)  Special  instructions. 

(i)  Relationship  of  inputs  to  outputs. 

(2)  Results  of  Operation.  A definition  of  expected  results  after 
completion  of  test  program  operation. 

(a)  Description  of  results. 

(b)  Form  in  which  results  will  appear. 

(c)  Output  format  and  content. 

(d)  Instruction  on  use  of  outputs. 

(e)  Limitations/restrictions. 

(f)  Relationship  of  outputs  to  inputs. 

(g)  Examples. 

f.  Operating  Instructions.  Procedures  required  to  operate  the  test  program. 

It  shall  include  the  following: 

(1)  Operating  procedures.  The  step-by-step  procedures  required  to: 

(a)  Initiate  the  test  program.  Procedures  shall  include  reading 
the  test  program  into  the  computer,  establishing  the  required 
mode  of  operation  (if  more  than  one),  initially  setting 
required  parameters,  providing  for  inputs  and  outputs,  and 
operating  the  test  program. 
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(b)  Maintain  test  program  operation.  Procedures  shall  be  specified 
to  maintain  operation  of  the  test  program  where  operator  inter- 
vention is  required. 

(c)  Terminate  and  restart  the  test  program.  Procedures  shall  be 
specified  for  normal  and  unscheduled  termination  of  test  program 
operation,  as  well  as  restarting  the  test  program. 

(d)  Software  system  generation  procedures.  This  section  shall 
include  both  the  routine  intervention  required  and  such  special 
topics  as  procedures  to  enable  operators  and/or  programmers  to 
"octal-in"  corrections  or  to  change  the  test  program  without 
recompiling  the  program  or  building  a new  system  master  tape. 
Procedures  should  also  be  included  that  are  necessary  to  create 
a new  version  of  the  master  code  media  (for  example,  master 
tape ) . 

(e)  Symbolic  updating  procedures.  This  section  shall  describe  the 
procedures  to  be  followed  to  effect  symbolic  changes  to  the 
software  system.  This  discussion  should  present  a step-by-step 
description  of  the  updating  processes.  As  necessary,  appro- 
priate data  sets  and  inputs  will  be  described. 

(2)  Operator  inputs.  A complete  description  of  all  the  control  cards  and 
card  formats  of  the  test  program,  including  the  function  or  purpose 
of  each  field,  will  be  given.  All  inputs,  other  than  those  described 
in  paragraph  le(l),  required  to  operate  the  test  program  will  be 
defined  in  a similar  manner  as  follows; 

(a)  Title  of  input. 

(b)  Purpose  and  use. 

(c)  Input  media. 

(d)  Limitations/restrictions. 

(e)  Format  and  content. 

(3)  Operator  outputs.  A complete  description  of  the  output  formats, 
other  than  those  described  in  paragraph  le(2),  of  the  test  program. 
This  shall  include  samples  of  each  type  of  possible  output  format. 
Furthermore,  a cross-reference  list  between  output  fields  and  input 
control  card  fields  will  be  given,  so  that  the  test  program  user  may 
readily  determine  the  effect  of  certain  input  fields  on  each  output 
field.  This  subparagraph  shall  include,  but  not  be  limited  to,  the 
following: 

(a)  Title  of  output. 

(b)  Purpose  and  use. 

(c)  Output  media. 

(d)  Output  format. 

(e)  Output  content  (symbols,  codes,  etc.). 
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g.  Appendix ■ The  appendix  may  include  information  bound  separately  for 
convenience,  as  in  the  case  of  classified  appendix  or  a large  body 
of  data. 

2.  Concept . Manual  content  and  format  shall  be  specifically  designed  to  meet 
the  needs  of  the  intended  user.  The  following  principles  further  explain 
the  users  manual  concept; 


a.  Each  manual  shall  be  organized  to  explain  the  system  in  terms  of  applica- 
tion and  operation. 

b.  Each  manual  shall  be  as  self-contained  as  possible.  Reference  to  other 
documents  should  beiminimal. 

c.  The  level  of  comprehension  for  each  manual  shall  be  appropriate  for  the 
intended  user  of  the  manual.  When  different  kinds  of  users  are  involved, 
the  manual  shall  be  written  so  that  all  users  can  comprehend  the  contents. 
Skill  level  5 shall  determine  the  base  comprehension  level  unless  other- 
wise specified. 

d.  Te.xt  shall  be  factual,  concise,  specific,  clearly  worded,  and  illustrated. 
Sentence  form  shall  be  simple  and  direct.  Abbreviated  tabular  data  such 
as  charts,  tables,  checklists  and  diagrams  shall  be  employed,  whenever 
practicable,  in  lieu  of  written  text. 

e.  Technical  knowledge  reflected  in  the  manual  shall  be  converted  into  the 
most  easily  understood  wording  possible.  Discussions  of  theory  shall 
be  omitted  except  where  essential  for  practical  understanding  and 
application.  Phraseology  requiring  a specialized  knowledge  shall  be 
avoided,  except  where  no  other  wording  will  convey  the  intended  meaning. 

The  primary  emphasis  will  be  placed  upon  the  specific  steps  to  be 
followed,  the  results  which  may  be  expected  or  desired,  and  the  correc- 
tive measures  required  when  such  results  are  not  obtained. 

3 . Production  Requirements: 

a.  Users  manuals  shall  be  designed  for  economy  of  initial  production  with 
respect  to  time.  This  implies  use  of  a standardization  format.  Basic 
format  shall  be  consistent  between  and  within  chapters  in  the  manual  so 
that  mass  production  data  preparation  techniques  may  be  employed. 

b.  Users  manuals  are  designed  for  economy  of  production  with  respect  to 
maintenance.  This  implies: 

(1)  Modular  construction.  Functions  shall  be  kept  separate  within  text 
to  facilitate  maintenance  procedures. 

(2)  Minimal  sequencing.  Unnecessary  numbering  of  sections,  paragraphs, 
tables,  and  figures  is  to  be  avoided  within  the  manual  so  that  periodic 
revisions  can  be  accomplished  with  a minimum  of  related  sequence 
changes  to  the  manual. 
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c . 


Relaxed  format  and  reproduction  methods  shall  be  permitted  for  any 
material  that  must  be  prepared  by  the  contractor  or  agency  furnishing 
the  manual.  Areas  which  may  be  relaxed  are  as  follows: 


(1)  Type  continuous  across  the  page. 

(2)  Use  of  standard  typewriter  to  prepare  reproduction  copy. 

(3)  Use  of  freehand  lettering  on  illustrations. 

(4)  Use  of  office-type  reproduction  equipment. 

(5)  Uniform  lettering  size  on  final  copy  not  required. 


d.  Illustration  procedures  shall  be  standardized  to  make  possible  the 

liberal  use  of  scripting  examples,  output  samples,  and  other  art  work. 


4.  Desi^nRequirements_.  The  primary  requirement  of  users  manual  design  is  that 

the  manual  shall  be  usable  by  the  intended  audience.  In  addition  the  follow- 
ing specific  requirements  shall  be  met: 


a.  Format . Each  document  shall  have  the  following  structural  parts:  a 

title  page,  a release  page,  a change  log,  a table  of  contents,  the  main 
body  of  text,  a glossary,  and  illustrations.  Optional  parts  are  a list 
of  figures  or  illustrations,  a list  of  tables,  appendixes,  f 1 an  index 
Each  page  of  the  manual  shall  contain  the  release  date,  pag  lumber, 
document  number,  and  page  content  heading. 


(1)  The  title  page  contains  the  title  of  the  manual,  the  contract  end 
item  number,  and  the  date  of  the  manual.  It  shall  be  prepared  under 
relaxed  format  style. 

(2)  The  release  page  contains  a description  of  the  version  of  the 
computer  program  system  with  which  the  users  manual  is  compatible. 

(3)  The  change  log  indicates  change  pages  and  shows  the  publication 
date  of  each  page. 

(4)  The  table  of  contents  contains  a list  of  topic  headings  taken  from 
the  main  body  of  the  text.  The  page  number  of  the  topic  headings 
shall  be  listed. 

(5)  The  optional  list  of  figures  or  illustrations  contains  a list  of 
the  titles  of  figures  or  illustrations.  The  page  numbers  of  the 
figures  or  illustrations  shall  be  listed. 

(6)  The  optional  list  of  tables  contains  a list  of  table  titles.  The 
page  numbers  of  the  tables  shall  be  listed. 

(7)  The  main  body  of  the  text  is  divided  into  chapters.  Each  main 
topic  shall  constitute  a chapter. 

(8)  The  glossary  shall  contain  all  specialized  terms  used  within  the 
manual . 

(9)  The  optional  appendixes  may  contain  any  auxiliary  material  deemed 
necessary  in  the  use  of  the  users  manual  or  the  computer  program 
system.  Examples  are  tables  and  worksheets. 


(10)  The  optional  index  contains  reference  page  numbers  of  each  topic 
listed . 
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b.  Pagination.  Pages  may  be  numbered  consecutively  throughout  the  document, 
or  through  a chapter  only. 

c.  Topic  Headings.  Topic  headings  shall  indicate  clearly  the  order  of 
subordination . Parts,  chapters,  sections,  paragraphs,  figures,  and 
tables  shall  have  brief  descriptive  titles.  Major  headings  may  be 
centered.  Subordinate  topic  headings  shall  be  left- justified  on  the 
page.  Run-in  headings  may  be  used  if  further  subordination  is  required. 

d.  Illustrations  and  Diagrams.  The  illustrations  and  diagrams  for  users 
manuals  shall  be  prepared  under  relaxed  format  style.  Illustrations  and 
diagrams  shall  be  used  in  lieu  of  text  whenever  this  technique  will 
result  in  a more  effective  presentation  of  information. 

e.  Nomenclature . Nomenclature  used  shall  be  consistent  throughout  a 
particular  set  of  users  manuals.  Standard  acron3nns  and  abbreviations 
may  be  used  provided  they  are  first  defined  in  the  text.  They  shall 
also  be  defined  in  the  glossary. 

f.  Space  Conservation.  Layout  shall  not  constrain  usability  or  clarity 
of  material.  If  white  space  increases  communication  effectiveness, 
blank  portions  of  pages  shall  be  permitted. 

g.  Users  Aids.  Summaries  and  printed  tables  shall  be  provided  where 
appropriate  to  aid  the  user  of  the  manual. 

h.  Collating,  Drilling,  and  Binding.  Collating,  drilling,  and  type  of 
covers  shall  be  as  directed  by  the  procuring  activity  and/or  user  agency. 
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SELF-TEST  DESIGN  ANALYSIS  FOR  SUPPORT  EQUIPMENT 


s.  ecaeiii»TtoN/»uii*esK 

To  establish  the  design  concepts  by  which  the  seller 
plans  to  support  and  implement  self  test  requirements. 
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This  data  item  is  utilized  to  obtain  the  analysis 
for  the  Automatic  Test  Equipment. 
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I.  Describe  rationale  used  in  mechanization  of  the  equipment's  self  test 
capabilities . 

II.  Provide  self  test  design  information  for  the  following  areas: 

A.  Identify  and  describe  in  detail: 

1.  Self  test  operating  modes 

2.  Equipment  failure  modes  and  relative  failure  rates 

(a)  All  WRA  failure  modes,  differentiating  Interface  Unit  (lU) 
failure  modes  from  others  associated  with  the  equipment. 

(b)  Failure  modes  affecting  safety. 

(c)  Failure  modes  to  be  detected  by  self  test. 

3.  Each  manual  non-automatic  test  operation  (if  subsystem  contains 
man/machine  interface). 

4.  Any  time  - dependent  constraints  to  testing. 


DO  ■^>1664  S/N  010>  0019*  4000 


*u*.«.a.e.i  i»Ti  MXii/raii 

96 


••••IT 


Page  2 of  2 


SELF-TEST  DESIGN  ANALYSIS  FOR  SUPPORT  EQUIPMENT  (Continued) 

Preparation  Instructions 

B.  Provide  and  discuss: 

1.  Schematic  diagrams  showing  implementation  of  self  test,  self  test 
circuit  isolation  and  test  point  locations. 

2.  Block  diagrams  showing  self  test  functional  areas  and  signal  flow. 

3.  Calculations  of  fault  detection  capabilities  (percent  assurance). 

4.  Calculations  of  fault  isolation  capabilities  (percent  certainty). 

C.  Design  approach  to  achieve  a 95%  fault  detection  capability  and  95%  fault 
isolation  certainty. 

1.  The  95%  probability  shall  be  calculated  based  on  the  weighted  failure 
rates . 

NOTE:  Computer  programming  shall  be  utilized  to  the  maximum 

extent  possible  for  performance,  evaluation  and  identification 
of  self  test  operational  modes  and  failures.  Detection  and 
isolation  information  may  be  used  by  the  computer  program  to 
provide  regressive  modes  of  operation. 

D.  Describe  the  self  test  approach  as  required  above  and  discuss  the  impact 
of  the  self-test  design  on  the  hardware  part  count,  weight,  and  volume. 

In  addition,  identify  the  percentage  of  weight  which  is  attributed  to 
self  test. 
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TEST  PROGRAM  FORMAT  REQUIREMENTS  (SOFTWARE) 


S.  OCACniATIOM/AURAOtC 


4.  AAANOVM.  OATK 


To  define  the  requirements  for  the  format  of  deliverable 
test  programs. 
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7.  AAPblCAriOM/IHTaRIICk  ATIOMAMIA 

These  data  requirements  apply  to  all  test  programs 
delivered  by  the  seller.  Test  Program  Flowcharts 
Standards  shall  be  as  specified  in  the  test 
instructions  and  shall  be  delivered  as  part  of 
this  data  item.  Test  Program  Coding  Standards 
shall  be  as  specified  in  the  TRD  and  shall  be 
delivered  as  required  as  part  of  this  data  item. 
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MIL-STD  1519 
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AUSRAMATIOM  IIMTRUCTI9NS 

1.  Test  Requirement  Documents  (TRD's)  shall  be  prepared  in  accordance  with 
MIL-STD- 15 19 

2.  Program  listings  shall  be  delivered  for  each  individual  routine  furnished. 
Each  listing  shall  be  prepared  in  the  ATLAS  language. 

3.  Test  programs  shall  have  appropriate  flowcharts  as  specified  in  computer 
Software  Flowchart(s)  Standards,  and  shall  meet  the  requirements  of  the 
TRD. 


4.  Source  programs  shall  be  furnished  in  card  image  on  magnetic  tape  and  as 
punched  card  decks.  Preparation  of  source  cards  shall  conform  to  Computer 
Software  Coding  Standards. 

5.  Object  programs  shall  be  prepared  in  machine  language  and  submitted;  a)  on 
magnetic  tape;  b)  as  punched  card  decks. 


6.  All  program  tables  and  files  necessary  for  the  successful  execution  of  programs 
shall  be  delivered  in  the  formats  required  by  the  TRD's.  They  shall  be  sub- 
mitted on  magnetic  tape,  card  deck  or  drum. 
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PROGRAMMER'S  NOTEBOOK 


a.  oc*eHi#rioM/»OMAe*B 


This  notebook  is  used  to  provide  Navy  personnel  with  the 
necessary  information  that  is  an  extension  to  the  pro- 
gramming manual.  The  notebook  further  explains  the  use 
of  instructions  or  sequences  of  instructions  for  a par- 
ticular type  of  function.  The  notebook  will  be  compiled 
by  the  lead  programmer  for  each  test  station. 
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T.  AVWieATIOM^IMTtMnCk  ATIONSMIP 

The  basis  for  the  notebook  will  be  the  experiences  and 
special  programming  tools  delivered  by  the  programmers 
as  a result  of  their  experiences  on  the  test  stations. 
The  notebook  will  be  submitted  with  acceptance  of  each 
station. 
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M.  IMSTIlUCTIOMt 

1.  Content . The  programmer's  notebook  shall  be  an  informal  loose  leaf  notebook 
maintained  by  the  lead  programmer  for  each  test  station.  It  shall  be  used  to 
document  experience  in  programming  the  computer  and  test  station.  A separate 
notebook  shall  be  delivered  with  each  test  station  upon  acceptance  of  the  test 
station. 

a.  Introduction/Nomenclature 

A description  of  the  station,  station  serial  number  and  other  identifying 
nomenclature . 

b . Glossary 

Terms  peculiar  to  this  particular  test  station.  Actions  that  are  peculiar 
to  the  individual  test  station. 

c . Test  Station  Programming  Extensions 

(1)  Areas  of  Extensions. 

(2)  Description  of  Extensions. 

(3)  Methods  of  use. 

(4)  Limitations. 

(5)  Implementing  Procedures. 

(6)  Operating  Instructions. 

2.  Concept . Notebook  content  and  format  shall  be  specifically  designed  to  meet  the 
needs  of  the  intended  user  - the  test  station  test  programmer.  Reference  to 
other  documents  should  be  minimal. 
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IOKNTIFICATION  noisi. 


SYSTEM/ DESIGN  TRADE  STUDY  REPORTS 


a.  ecscMivTiON/AUMPoa 


re  A*»«OVAt.  OATC 


Provides  data  relating  to  design  decisions  including 
BIT  level,  cost,  maintenance  policy,  test  point  selection, 
maintainability  and  other  tradeoffs. 
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Used  to  determine  basis  for  design  decisions  and 
maintenance  strategy. 


1a.  ncrcKKNeca  (mnOaiorr  ••  c«w  m 
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AR  lOA,  21,  30A 
NAVMATINSTR  3960.0 
MIL-STD  1304(AS) 
MIL-STD  1390A 
MIL-STD  1388 
MIL-STD  847 
MIL-STD-1519 
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10.  AHBaARATION  IMaTRUCTIONa 

Submit  details  and  results  of  tradeoff  study  reports  including  determination  of: 

o BIT  level  (detection  and  isolation) 
o Maintenance  policy 

o Testability  level 

o Reliability 

o Maintainability 

o Hardware  and  software  growth  due  to  BIT 

o ATE  selection 

o Adapter  and  interface  configurations 

10.1  Unless  otherwise  indicated  herein,  documents  cited  in  this  block  shall  be  of 
the  issue  in  effect  on  the  date  of  invitation  for  bids  or  request  for  proposals  or 
quotations  and  shall  form  a part  of  this  Data  Item  Description  to  the  extent 
specified  herein. 

10.2  Format . This  report  shall  be  prepared  in  accordance  with  MIL-STD-847. 

10.3  Content . This  report  shall  contain  the  design  trade-offs  conducted  as  part 
of  the  Reliability  and  Maintainability  Program  to  ensure  testability  and  support- 
ability  of  the  end  item.  Analyses  to  prepare  test  requirements  data,  procedures 
and  guidelines  to  be  used  in  this  effort  are  described  in  the  Naval  Material 
Command  (MAT-04T)  documents  entitled  "Acquisition  Planning  Guide  for  Automatic 
Test,  Monitoring  and  Diagnostic  Systems  and  Equipment"  dated  1 July  1976  and 
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10.3  Content  (Continued) 

"Built-In-Test  (BIT)  Design  Guide"  dated  1 July  1976.  The  contractor  shall  utilize 
the  above  cited  documents  in  the  acquisition  process  of  platforms,  systems  and 
equipment  (end  items)  to  provide  assurance  of  early  identification  of  ATE  require- 
ments, proper  definition  of  the  requirements  in  contractual  documents,  and 
systematic  appraisal  and  implementation  of  ATE  requirements  during  design  and 
development  of  the  end  item.  The  results  should  provide  for  a more  operationally 
effective  and  supportable  weapon  system  at  an  acceptable  life  cycle  cost. 

a.  The  Test  Requirements  Analysis  (TRA)  is  to  be  performed  at  both  the 
conceptual  (System)  and  Development  (Equipment)  phase  during  normal  program 
evolution.  At  the  conceptual  phase,  the  TRA  serves  as  a vehicle  by  which  opera- 
tional and  system  requirements  evolve  a preliminary  ATE  support  concept.  This 
concept  is  formulated  in  an  iterative  system  engineering  design  process  and  it 
provides  goals  and  objectives  to  the  equipment  designer  for  him  to  perform 
detailed  WRA  (Weapons  Replaceable  Assembly)  and  SRA  (Shop  Replaceable  Assembly) 
development  on-line  with  the  system  maintenance  concept.  The  preliminary  or  System 
Test  Requirement  Analysis  conducted  during  the  conceptual  phase  serves  as  an 
input  into  the  system  design  and  Integrated  Logistics  Support  Process. 

b.  At  the  conclusion  of  the  validation  phase  of  program  development, 
candidate  UUTs  (Unit-Under-Test)  for  off-line  testing  should  be  identified 
based  on  Level  of  Repair  Analysis  (LORA),  Logistic  Support  Analysis  (LSA) , and 
design  trade-off  analysis.  Detailed  design  information  for  these  candidate 
UUTs  should  then  be  utilized  as  source  documentation  for  Equipment  Test  Require- 
ment Analysis.  This  analysis  should  be  performed  as  per  MIL-STD-1519  with  excep- 
tion/modifications as  required  and  approved  by  NAVAIR.  The  Equipment  TRA  when 
executed  per  MIL-STD-1519  manifests  itself  in  a normalized  UUT  data  base  end  item 
called  the  Test  Requirements  Document  (TRD).  This  document  serves  as  a point 
source  vehicle  for  all  performance  verification  and  diagnostic  procedures  and 
equipment  requirements  to  support  the  WRA  in  its  maintenance  environment,  whether 
supported  manually  or  via  ATE. 

I 

c.  In  addition.  System  and  Equipment  Test  Requirement  Analyses  may  also  be 
utilized  to  define  and  refine  the  quantity  and  type  (i.e..  Manual  or  Automatic) 
of  off-line  equipment  required  to  support  Avionic  Equipment  suits  and  associated 
GSE  (Ground  Support  Equipment)  as  a system. 

d.  The  contractor  shall  submit/update  this  report  and  content  as  appro- 
priate to  each  phase  of  development  (i.e.,  for  the  conceptual,  validation,  full 
scale  development  or  production)  specified  by  the  contract. 

I 

1 
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Page  3 of  4 

SYSTEM/DESIGN  tRADE  STUDY  REPORTS 
Preparation  Instructions  (Continued) 

10.3  Content  (Continued) 


e.  The  system  TRA  shall  describe  the  System  Level  Maintenance  Strategy  at 
the  "0"  Level  plus  set  "Design  to  Requirements"  for  the  equipment  to  be  tested 

at  the  "I"  and  "D"  Level  as  well.  The  system  TRA  should  concentrate  on  descrip-  i 

tion  of  test  methods,  organization  of  system  architecture,  determination  and  i 

location  of  individual  equipment  test  points,  and  create  the  documentation  base- 
line necessary  to  support  and  describe  the  maintenance  philosophy  of  the  system. 

A logical  sequence  of  data  development  and  the  manner  in  which  data  should  be 
reviewed,  verified,  and  utilized  must  also  be  stipulated  and  established  in  the 
System  Level  TRA.  A preliminary  TRA  should  be  provided  as  part  of  the  contractor's 
original  proposal  during  a program's  Conceptual  Phase  and  should  be  subsequently 
updated  as  the  design  progresses  and  upon  conclusion  of  the  program's  Validation 
and  Full  Scale  Development  Phases. 

f.  The  contractor  shall  describe  how  he  plans  on  validating  his  System  j 

Maintenance  Concept  at  the  Organizational  Level.  This  validation  should  take  the 

form  of  a BIT  fault  detection/isolation  (F/I)  demonstration  performed  on  an  j 

Advanced  Development  Vehicle  during  the  Validation  Phase  of  program  development. 

The  validation  plan  should  stipulate  how,  when,  and  where  this  demonstration  is 
to  be  performed.  The  demonstration  shall  exercise  the  Operational  Readiness  i 

Test  to  validate  the  testability,  maintainability,  and  supportability  features  j 

of  the  equipment  designed.  j 

I 

g.  The  equipment  TRA  shall  describe  the  functional  end-to-end  (E/E)  test  i 

requirements  and  fault  isolation  test  procedures  required  for  each  UUT  to  be  j 

consistent  with  the  Maintenance  Concept  stipulated  in  the  System  Level  TRA. 

Equipment  TRAs  should  be  performed  in  accordance  with  MIL-STD-1519  and  submitted 
for  each  UUT  in  two  major  stages.  A Preliminary  or  End-to-End  TRA  should  be  pro- 
vided for  each  UUT  identified  by  the  contractor  as  an  Off-Line  Candidate  prior  to 
the  conclusion  of  the  Validation  Phase  of  Program  Development.  A more  detailed 
TRA  encompassing  both  an  up-date  of  the  preliminary  E/E  Functional  test  require- 
ments, Fault  Isolation  Procedures  and  test  techniques  should  be  submitted  during 
the  Full  Scale  Development  Phase.  The  TRA  shall  be  kept  current  throughout  the 
Life  Cycle  of  the  individual  equipments  being  procured. 

h.  The  contractor's  off-line  ATE  selection  shall  be  detailed  to  show  the 
proposed  solution  of  an  optimum  ATE/Manual  Support  Vehicle/Vehicles  per  the 
considerations  stipulated  in  the  Acquisition  Planning  Guide  for  Automatic  Test, 

Monitoring  and  Diagnostic  Systems  and  Equipment.  Primary  consideration  should 
be  given  to  existing  GFE  (Government  Furnished  Equipment)  ATE  Assets  in  lieu  of 
new  systems.  (Applicable  to  Full  Scale  Development  Phase). 
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SYSTEM/DESIGN  TRADE  STUDY  REPORTS 
Preparation  Instructions  (Continued) 

10.3  Content  (Continued) 

i.  The  contractor's  off-line  F/I  demonstration  plan  shall  describe  the 
plans  validating  the  Intermediate  and/or  Depot  Level  Maintenance  Concept.  The 
validation  should  take  the  sum  of  individual  fault  isolation  demonstrations 
performed  on  Full  Scale  Development  UUTs  utilizing  ATE  and/or  manual  test 
equipment.  (Applicable  to  Full  Scale  Development  Phase). 

j. '  Test  Program  Set  (TPS)  elements  required  to  test  and  fault  isolate  a 
UUT  Off-Line  should  be  generated  upon  conclusion  of  an  individual  UUT's  F/I 
demonstration.  This  data  should  include  but  is  not  restricted  to:  (1)  Test 
Program  Tape  or  Cassette,  (2)  Test  Program  Instruction,  (3)  Any  Interconnection 
Devices  or  Interface  Hardware,  (4)  Final  Test  Requirements  Document,  and 

(5)  Test  Diagrams.  (Applicable  to  Full  Scale  Development  Phase). 

k.  The  contractor's  test  program  set  fault  isolation  demonstration  results 
shall  describe  the  following  for  each  UUT  Fault/Isolation  demonstration  performed: 

(1)  Fault  names  inserted  referenced  to  applicable  documentation  and  schematics, 

(2)  Results  of  each  fault  inserted  and  where  applicable  reason/reasons  why  fault 
was  not  isolated  properly,  and  (3)  Detailed  plans  as  to  corrective  action  to  be 
taken  on  all  undetected  faults  or  false  diagnostics  incurred  during  the  F/I 
demonstration.  (Applicable  to  Full  Scale  Development  Phase). 

l.  The  contractor's  TPS/TRD  UPDATE  PLAN  shall  describe  how  he  plans  to 
upgrade  individual  Test  Program  Sets  and  TRDs  based  upon  the  F/I  demonstration 
and  Unsatisfactory  Reports  (URs)  from  the  field  activities.  (Applicable  to 
Full  Scale  Development  Phase). 

10.4  Changes.  Whenever  changes  to  the  end  item  are  made  or  contemplated,  the 
contractor  shall  provide  revision  material  for  this  report  to  reflect  the  change. 
Revised  material  shall  bear  the  same  page  numbers  as  those  pages  which  are  to 

be  replaced,  plus  the  word  "revised"  and  the  date  of  revision.  Additional  pages 
shall  bear  the  same  page  number  as  the  preceding  page  followed  by  a lower  case 
letter  unless  the  additional  pages  follow  the  last  page  of  the  report.  Revised 
or  added  material  shall  include  a revised  title  page  indicating  the  date  of  the 
revision.  The  revised  title  page  shall  contain  the  information  contained  on  the 
original  title  page  plus  a revised  index  of  revisions. 
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The  Test  Program  Set  Configuration  Management  Plan 
defines  the  Contractor's  policies  and  procedures  for 
implementing  the  requirements  specified  in  the  Configura- 
tion Management  Specification. 


7.  A»#klCAriOM/IMT>IIMSk  ATIONIMIV 


The  Test  Program  Set  Configuration  Management  Plan 
shall  describe  the  methods  to  be  used  by  the  contractor 
to  implement  the  Configuration  Management  Specification. 
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1.  The  Test  Program  Set  Configuration  Management  Plan  shall  describe  the  methods 
by  which  compliance  with  the  Configuration  Management  Specification  will  be 
accomplished. 

2.  The  plan  shall  not  be  used  to  alter,  modify  or  supersede  the  CM  Specification 
requirements . 

3.  MIL-STD-483,  Appendix  I may  be  used  for  additional  guidance. 

4.  The  format  requirements  shown  below  apply. 
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CONFIGURATION  MANAGEMENT  PLAN  FORMAT 


SECTION  I 


SECTION  II 


SECTION  III 


SECTION  IV 


SECTION  V 


SECTION  VI 


GENERAL 

A.  Introduction 

B.  Compliance  and  Precedence 

C.  Organization  and  Responsibilities 
IDENTIFICATION  CONTROL 

A.  Specifications 

B.  Drawings 

C.  Equipment/Hardware/Software 

D.  Release  Records 
CONFIGURATION  CONTROL 

A.  Baselines 

B.  Change  Control 

C.  Deviations  and  Waivers 
CHANGE  VERIFICATION  SYSTEM 
A.  Verification  Records 
CONFIGURATION  MANAGEMENT  AUDITS 

A.  Configuration  Management  Systems  Audits 

B.  Physical  Configuration  Audit  (PCA) 
SUBCONTRACTOR/SUPPLIER  AUDITS 
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lOCMTIPICATIOM  NO(S>. 


AOCNCV 


NUMMN 


I.  TITkC 

EXCEPTION  TO  STANDARD  ELECTRONIC  MODULES  (SEM) 


*.  Oi 


4.  4»riteVAC  04TC 


Requests  for  exception  to  SEM  requirements  provides 
technical  and  economic  justification  by  the  supplier  in 
order  that  SEM  requirements  may  be  waived  by  the  buyer. 
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Requests  for  exception  to  SEM  requirements  are  initiated 
by  the  supplier  when  SEM  requirements  (existing  modules 
and/or  new  module  design  requirements)  are  determined  to 
be  unsuitable  for  system  hardware  requirements  or  portions 
thereof. 


•.  nercito 

6toc*  ic: 


MIL-STD-1378 


\ m in 


MCtL 


«o.  nncPAAArtoN  imstmuctioms 

1.  Requests  shall  be  in  the  form  of  written  letter  reports,  providing  the 
supplier's  justification  for  a deviation  or  waiver  to  specific  SEM  requirements. 
In  instances  where  cost  effectiveness  is  the  basis  for  exception,  the  cost 
model  information  shall  be  prepared  by  the  supplier  in  accordance  with  pro- 
visions of  the  Cost  Effectiveness  Model  of  MIL-STD-1378  in  addition  to  other 
applicable  items  of  the  format  below. 

2.  The  report  shall  contain  as  a minimum,  the  following: 

a.  Contractor  Name. 

b.  Contractor  Number. 

c.  Nomenclature  or  description  of  major  equipment  component. 

d.  Identification  of  part  and/or  the  identification  of  the  technical  require- 
ment under  consideration. 

e.  Technical  viewpoints,  pro  and  con  with  justification. 

f.  Economic  viewpoints,  pro  and  con  with  practical  cost/expenditure  formulation 
and  justification. 

g.  Alternate  approaches  to  include  narrative  information,  illustrations, 
graphic  or  schematics  as  applicable. 

h.  . Tradeoffs  involving  patents'  or  proprietary  data  (rights  in  data)  if 

applicable . 

i.  Name  of  engineer  or  representative  preparing  report,  his  telephone  number 
and  mailing  address. 

j.  Signature  of  company  official. 
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Preparation  Instructions  (Continued) 


3.  The  above  cited  document,  of  the  issue  in  effect  on  the  data  of  invitation  for 
bids  or  request  for  proposal,  form  a part  of  this  Data  Item  Description  to  the 
extent  specified  herein. 
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POLICIES  AND  PROCEDURE  ANALYSIS 


CONCEPTUAL  PHASE 
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1.0  INTRODUCTION 

This  report  is  the  first  of  four  reports  to  be  generated  as  part  of  the  policies  and 
procedures  task  item.  The  object  of  this  report  is  to  provide  a comprehensive  work 
breakdown  structure  (WBS)  for  ATE  related  activities  to  be  undertaken  during  the 
conceptual  phase  of  program  development  for  Navy  weapon  systems.  This  work 
breakdown  structure  will  provide  a baseline  for  the  review,  analysis,  modification,  and 
generation  of  policies  and  procedures  germane  to  the  ATE  process  during  the 
conceptual  phase  of  program  development.  This  WBS,  when  integrated  into  the 
Acquisition  Planning  Guide,  will  ultimately  provide  a consistent  framework  for: 

o Assisting  project  personnel  in  planning  and  assigning  responsibilities  for 
ATE  related  activities  during  the  conceptual  phase  of  program 
development. 

o Assisting  project  personnel  in  controlling  and  reporting  progress  on 
conceptual  phase  trade  studies  and  support  activities. 

o Establishing  a data  base  for  estimating  the  support  costs  for  a Navy  weapon 
system  during  the  conceptual  phase  of  program  development. 

The  basis  for  this  report  is  the  Conceptual  Phase  portion  of  Figure  1-1  (Functional 
Flow  of  Acquisition  Activities)  of  the  Acquisition  Planning  Guide  for  Automatic  Test, 
Monitoring  and  Diagnostic  Systems  and  Equipment.  The  conceptual  phase  work 
breakdown  structure,  as  depicted  by  this  document,  is  shown  in  Figure  G-1.  Figure  G-2 
depicts  an  augmented  WBS  derived  as  a result  of  studies  conducted  under  the  policies 
and  procedures  task  item. 

2.0  CONCEPTUAL  PHASE  WBS 

Comparison  of  Figures  G-1  and  G-2  indicates  that  the  primary  differences  in  the 
WBS  is  in  the  area  of  ATE  trade-off  analysis.  With  this  in  mind,  the  following 
discussion  is  limited  to  this  aspect  of  the  conceptual  phase  WBS. 

Prior  to  the  publication  of  the  ATE  acquisition  guide,  no  suitable  tool  existed  for 
performing  test  equipment  trade-off  analysis.  Subsequently  since  its  release,  the  test 
equipment  effectiveness  model  (TEEM)  has  come  upon  the  scene.  The  test  equipment 
effectiveness  model  was  developed  by  the  Naval  Weapons  Engineering  Support  Activity 
(ESA-8)  under  sponsorship  of  Headquarters,  Naval  Material  Command  (MAT  04T). 

The  model  is  a computerized  tool  which  can  assist  project  personnel  in 
determining  the  mix  of  test  equipment  best  suited  to  support  an  end-item.  The  user 
must  supply  deployment,  prime  equipment,  and  test  equipment  data  to  the  model.  The 
model  utilizes  this  data  together  with  internal  default  data  (i.e.,  shipping  costs, 
facility  overhead  costs)  to  provide  estimates  of  the  following  parameters: 

o Prime  system  availability  (projected) 

o Quantities  of  support  equipment  required  per  site 
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o Test  personnel  required  per  site 

o Queueing  estimates 

o Recurring  and  non-recurring  costs  per  test  equipment  site 

o Failure  rate  sensitivity  analysis. 

The  model  is  used  by  selecting  several  test  scenarios  and  supplying  the  pertinent  data 

for  each.  Manual  comparisons  are  then  made  of  the  resultant  output  estimates  to 
select  the  best  of  the  alternate  approaches  used.  Of  itself,  the  model  does  not 
optimize  between  alternate  approaches.  The  model  does  not  satisfy  the  requirements 
for  a level  of  repair  (LORA)  or  life  cycle  cost  (LCC)  analysis.  It  is  instead  a 
supplement  to  these  analyses.  Generalized  repair  versus  discard  decisions  must  be 
made  prior  to  using  the  model.  The  model  provides  the  following  advantages: 

o Rapid  analysis  and  cost  prediction  for  comparative  evaluation  of 
maintenance/support  philosophies. 

o Cost  sensitivity  analysis  vice  prime  equipment  removal  rate. 

o Simplified  input  format,  tailored  to  the  level  of  data  available  in  a program 

planning  stage. 

At  present,  the  model  has  the  following  limitations: 

o Inability  to  handle  real-world  situations,  in  which  a mix  of  test  equipments 
may  be  proposed  at  each  test  site. 

o Inability  to  perform  a queueing  analysis  for  more  than  one  UUT  across  a 
tester.  That  is,  perform  calculations  which  steer  one  to  the  "optimum" 
support  vehicle. 

o Inability  to  allow  for  interaction  with  the  user,  for  example,  in  modifica- 
tion of  the  default  values.  For  this,  a programmer  is  required  to  alter  the 
model  source  deck. 

o Inability  to  consider  the  costs  of  BIT,  within  the  model,  as  contributing  to 
support  costs. 

Conceptually,  the  above  limitations  may  be  overcome  by  the  user  if  he  is  familiar 
enough  with  the  mathematics  of  the  model  to  alter  his  source  data,  or  analyze  the 
output  data,  to  compensate  for  model  performance  (in  effect,  to  trick  the  model  into 
doing  the  right  thing). 


3.0  SHORT  TERM  RECOMMENDATIONS 


I 


The  following  short  term  recommendations  are  made  relative  to  the  TEEM 
model: 

o Solutions  and  cost  estimates  should  be  solicited  by  MAT  04T  from  the 
Naval  Weapons  Engineering  Support  Activity  (ESA-8)  to  accomplish  the 
following: 

Modify  or  add  a front  end  program  to  allow  the  TEEM  model  to  be 
used  in  a real  world  situation  (i.e.,  accommodate  several  testers  at 
each  level  of  test). 

Modify  the  TEEM  model  to  provide  a BIT  related  cost  factor. 

Modify  the  TEEM  model  to  permit  user  interaction  for  alteration  of 
assumed  or  default  values. 

Modify  the  TEEM  model  to  allow  queuing  analysis  for  more  than  one 
UUT  across  a given  tester. 

o After  the  above  mentioned  recommendations  have  been  implemented,  it  is 
recommended  that  the  TEEM  model  be  benchmarked  (i.e.,  validated) 
against  a real  life  test  scenario. 

o Obtain  resource  estimates  from  ESA  relative  to  providing  TEEM  support 
for  the  three  SYSCOMs. 

4.0  LONG  TERM  RECOMMENDATIONS 

The  following  long  term  recommendations  are  made  relative  to  two  major 
elements  (data  banks,  ATE  trade-off  analysis)  which  comprise  the  conceptual  phase 
WBS: 

o The  present  data  banks  covered  under  NAVMAT  Instruction  5230.8  are 
woefully  out  of  date  and  do  not  address  the  needs  of  the  user.  Write  a 
specification  (i.e.,  input/output  data  requirements  for  a data  bank  which 
could  be  used  throughout  the  ATE  selection  process  - from  very  general 
support  requirements  developed  in  the  conceptual  phase  to  the  finalization 
of  specific  test  requirements  in  the  production  phase.  The  subject  data 
bank  must  have  family  of  tester  data  relevant  to  the  needs  of  the  three 
SYSCOMs  and  also  satisfy  the  input  data  requirements  of  the  TEEM  model. 
Keep  the  data  bank  current.  Benchmark  the  data  bank  and  then  generate 
an  instruction  advertising  its  existence. 

o Generate  an  instruction  advertising  the  existence  of  the  TEEM  model  upon 
successful  completion  of  its  validation  by  an  independent  authority. 
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APPENDIX  H 


POLICIES  AND  PROCEDURES  ANALYSIS 
VALIDATION  PHASE 


1.0  INTRODUCTION 


This  report  is  the  second  of  four  reports  to  be  generated  by  ManTech  of 
New  Jersey  Corporation  as  part  of  the  policies  and  procedures  task  item.  The  object 
of  this  report  is  to  provide  a comprehensive  work  breakdown  structure  (WBS)  for  ATE 
related  activities  to  be  undertaken  during  the  validation  phase  of  program  development 
for  Navy  weapon  systems.  This  work  breakdown  structure  will  provide  a baseline  for 
the  review,  analysis,  modification,  and  generation  of  policies  and  procedures  germane 
to  the  ATE  process  during  the  validation  phase  of  program  development.  This  WBS, 
when  integrated  into  the  Acquisition  Planning  Guide,  will  ultimately  provide  a 
consistent  framework  for: 

o Assisting  project  personnel  in  planning  and  assigning  responsibilities  for 
ATE  related  activities  during  the  validation  phase  of  program  development. 

o Assisting  project  personnel  in  controlling  and  reporting  progress  on 
validation  phase  trade  studies  and  support  activities. 

o Establishing  a data  base  for  estimating  the  support  costs  for  a Navy  weapon 
system  during  the  validation  phase  of  program  development. 

The  basis  for  this  report  is  the  Validation  Phase  portion  of  Figure  1-1  (Functional  Flow 
of  Acquisition  Activities)  of  the  Acquisition  Planning  Guide  for  Automatic  Test, 
Monitoring  and  Diagnostic  System  and  Equipment.  The  validation  phase  work 
breakdown  structure  as  depicted  by  this  document  is  shown  in  Figure  H-1.  Figure  H-2 
depicts  an  augmented  WBS  derived  as  a result  of  studies  conducted  under  the  policies 
and  procedures  task  item. 

2.0  VALIDATION  PHASE  WORK  BREAKDOWN  STRUCTURE 

Comparison  of  Figures  H-1  and  H-2  indicates  that  the  primary  differences  in  the 
WBS  of  ATE  related  activities  lies  in  the  area  of  refining  ATE  application 
requirements.  With  this  in  mind,  the  following  discussion  is  for  the  most  part  limited 
to  this  aspect  of  the  validation  phase  WBS.  In  the  development  of  a modern  weapon 
system,  the  technology  of  the  elements  to  be  maintained  are  typically  more  advanced 
than  the  ATE  technology  available  to  support  these  end  items  off-line.  During  the 
conceptual  phase  of  program  development,  the  areas  of  advanced  technology 
development  contemplated  should  be  extracted  from  the  preliminary  prime  system 
performance  characteristics.  This  information  should  be  compared  with  current  ATE 
technology  and  voids  in  testing  technology  identified.  Upon  entering  the  validation 
phase  of  program  development,  (see  Figure  H-2)  unique  or  advanced  (new)  technology 
support  requirements  for  system  configuration  end  items  should  be  verified  and 
specified.  The  verification  process  consists  of  surveying  the  current  technology 
marketplace  and  verifying  that  the  end  item  in  question  requires  special  off-line 
support  considerations.  System  functional  (block)  diagrams  and  performance 
requirements  which  evolve  during  the  validation  phase  of  program  development  are  key 
tools  in  discerning  whether  advanced  technology  ATE  or  ATE  currently  available  in  the 
commercial  and  military  inventory  can  support  the  weapon  system  under  development. 
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FULL  SCALE 

DEVELOPMENT  ACTIVITIES 


Figure  H-2.  Validation  Phase  WBS  — Policies  and  Procedures  Task  Item 


An  objective  unbiased  technology  assessment  during  the  validation  phase  of  program 
development  is  a prerequisite  for  avoiding  underestimation  of  a weapon  system  support 
problem  which  will  become  quite  apparent  in  later  stages  of  program  development. 
Confirmation  that  an  advanced  technology  support  requirement  exists  is  a strong 
argument  for  the  inclusion  of  BIT  as  part  of  the  new  weapon  system  design  for  both 
RFl  (performance)  and  diagnostic  testing.  Where  this  approach  does  not  appear 
operationally  or  economically  feasible,  suitable  off-line  advanced  technology  support 
requirements  must  be  specified  and  planned  for.  In  either  case,  the  results  of  trade 
studies  which  mandate  adherence  to  the  support  concept  must  be  reflected  in  the 
revised  ATE  support  concept  for  the  prime  weapon  system. 

Advanced  technology  or  unique  support  requirements  should  be  documented  via  a 
support  equipment  specification.  A statement  of  work  and  associated  data 
requirements  should  also  be  prepared  for  submittal  with  the  RFP  for  acquisition  of 
advanced  technology  or  unique  ATE.  Prior  to  RFP  release,  the  planning  and 
acquisition  requirements  for  advanced  technology  ATE  should  be  established  and 
documented  in  the  off-line  ATE  support  plan  for  the  weapon  system  being  supported. 

As  indicated  in  Figure  H-2,  the  TEEM  model  should  be  utilized  during  the 
validation  phase  to  further  refine  the  ATE  support  concept  for  those  UUTs  that  appear 
to  be  compatible  with  present  day  ATE  technology.  Updated  deployment,  prime 
equipment,  and  support  system  data  (available  via  data  banks)  in  unison  with  a more 
finely  defined  prime  system  architecture  will  enable  TEEM  analysis  to  divulge  the 
optimum  system  support  strategy. 

Harmonization  of  the  delivery  of  advanced  technology  support  tools  with  readily 
available  off-line  ATE  selected  and  acquired  during  the  full  scale  phase  of  program 
development  is  highly  desirable.  However,  experience  has  shown  that  such  an 
occurrence  is  highly  unlikely.  Early  identification  of  advanced  testing  technology 
requirements  in  the  conceptual  phase  and  the  timely  verification  of  the  requirement 
and  specification  in  the  validation  phase  is  mandatory  to  ensure  support  system 
success.  Any  delay  incurred  in  contracting  and  developing  advanced  technology 
support  tools  during  the  validation  phase  will  have  an  adverse  effect  on  satisfying  test 
program  set  and  R&M  demonstration  requirements  required  in  the  full  scale 
development  phase  of  program  development.  This  effect  may  in  turn  have  an  adverse 
I effect  on  advanced  technology  ATE  deliveries  during  the  production  phase  of  program 

I development. 


3.0  SHORT  TERM  RECOMMENDATIONS  I 

When  conducting  program  reviews  and  consultation  under  the  MAT  04T  umbrella, 
require  that  a technology  assessment  of  proposed  configuration  end  items  versus  ATE 
technology  available  be  conducted  during  the  conceptual  and  validation  phases  of 
program  development.  As  an  example,  the  aspect  of  technology  assessment  is 
currently  being  pursued  in  the  RPV  program  via  close  coordination  between  AIR  53^^ 
and  ManTech.  Advanced  or  new  technology  configuration  end  items  for  this  particular  i 

program  should  be  identified  by  August  of  this  year.  | 
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4.0  LONG  TERM  RECOMMENDATIONS 


r 


Create  policies  and  procedures  which  would  assist  project  managers  in 
conducting  end  item  versus  ATE  technology  assessment  in  some  standard  or  normalized 
manner.  Such  a policy  should  be  based  on  the  marriage  of  UUT  performance  data 
versus  ATE  application  data  in  a data  bank.  The  output  of  the  data  bank  analysis 
should  be  unambiguous  - either  ATE  exists  in  the  current  inventory  to  support  the 
designated  end  item  or  it  doesn't.  If  the  data  bank  analysis  indicates  new  ATE  is 
warranted,  a LORA  analysis  would  have  to  be  performed  to  economically  justify  the 
utilization  of  the  proposed  new  ATE.  The  LORA  would  have  to  take  into  consideration 
both  the  recurring  and  non-recurring  development  and  operational  costs  for  the  new 
ATE  which  supports  the  configuration  end  item  in  question. 

5.0  REFERENCES 

(a)  Test  Equipment  Effectiveness  Model  User's  Guide,  U.S.  Naval  Weapons 
Engineering  Support  Activity  - Management  Engineering  Department, 
6 January  1977. 

(b)  Acquisition  Planning  Guide  for  Automatic  Test,  Monitoring  and  Diagnostic 
Systems  and  Equipment,  Test  and  Monitoring  Systems  Office  (MAT  04T), 
1 July  1976. 

(c)  Policies  and  Procedures  Analysis  - Conceptual  Phase  Work  Breakdown 
Structure,  Report  prepared  by  ManTech  of  New  Jersey  Corporation  for 
MAT  04T,  28  February  1977. 
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FULL  SCALE  DEVELOPMENT 
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1.0  INTRODUCTION 


F 


This  report  is  the  third  of  four  reports  to  be  generated  as  part  of  the  policies  and 
procedures  task  item.  The  objective  of  this  report  is  to  provide  a comprehensive  work 
breakdown  structure  (WB5)  for  ATE  related  activities  to  be  undertaken  during  the  full 
scale  development  phase  of  program  development  for  Navy  weapon  systems.  This  work 
breakdown  structure  will  provide  a baseline  for  the  review,  analysis,  modification,  and 
generation  of  policies  and  procedures  germane  to  the  ATE  process  during  the  full  scale 
development  phase  of  program  development.  This  WBS,  when  integrated  into  the 
Acquisition  Planning  Guide,  will  ultimately  provide  a consistent  framework  for: 

o Assisting  project  personnel  in  planning  and  assigning  responsibilities  for 
ATE  related  activities  during  the  full  scale  development  phase  of 
acquisition  process. 

o Assisting  project  personnel  in  controlling  and  reporting  design  and 
development  of  first  article  UUT  hardware  and  TP5s  and  ensure  proper 
selection  of  ATE  for  maintenance  support. 

o Establishing  a data  base  for  estimating  the  support  costs  for  a Navy  weapon 
system  during  the  full  scale  development  phase  of  program  development. 

The  basis  for  this  report  is  the  Full  Scale  Development  portion  of  Figure  l-I 
(Functional  Flow  of  Acquisition  Activities)  of  the  Acquisition  Planning  Guide  for 
Automatic  Test,  Monitoring  and  Diagnostic  System  and  Equipment.  The  full  scale 
development  breakdown  structure  as  depicted  by  this  document  is  shown  in  Figure  I-l. 
Figure  1-2  depicts  an  augmented  WBS  derived  as  a result  of  studies  conducted  under 
the  policies  and  procedures  task  item. 

2.0  FULL  SCALE  DEVELOPMENT  (FSD)  PHASE  WORK  BREAKDOWN  STRUCTURE 

Comparison  of  Figures  I- 1 and  1-2  indicates  that  the  primary  differences  in  the 
work  breakdown  structure  during  the  FSD  phase  are  in  the  areas  of  (1)  Test 
Requirements  Analysis  (TRA),  (2)  TRD  Management,  (3)  ATE  selection,  (4)  TPS 
generation,  (5)  Configuration  control  of  ATE,  TPS,  TRD  and  UUT  related  items. 

2.1  TRA.  TRA  is  a process  which  defines  the  functional  end-to-end  (E/E) 
performance  requirements  and  fault  isolation  (F/I)  test  procedures  which  are  required 
for  each  UUT  to  be  consistent  with  the  overall  system  maintenance  concept.  The 
input  to  TRA  process  is  the  UUT  design  data  which  is  used  during  the  analysis  to 
determine  test  requirements.  The  UUT  design  data  consists  of  drawings  (schematics, 
logic  diagrams,  parts  lists,  etc.),  failure  and  MTBF  data,  performance  specification, 
theory  of  operation  and  mechanical/electrical  interface  definition.  The  output  of 
equipment  TRA  process  is  recorded  in  appropriate  sections  of  the  TRD  document  for 
E/E  and  F/I  test  requirements.  These  functions  are  normally  performed  by  the  UUT 
designer  having  most  familiarity  with  the  unit,  however,  in  a case  of  simple  modules 
and  throwaways,  these  functions  can  also  be  performed  by  other  activities,  such  as  TPS 
developer. 
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FULL  SCALE  DEVELOPMEMT  PHASE 


Figure  1-1.  Full  Scale  Development  Phase  WBS  — Acquisition  Planning  Guide  for 
Automatic  Test,  Monitoring  and  Diagnostic  Systems  and  Equipment. 
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Figure  1-2.  Full-scale  Development  Phase  WBS  Policies  and  Procedures  Task  Item. 
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2.2  TRD  management.  The  purpose  of  the  TRD  specification  is  to  identify  the 
necessary  data  for  development  of  cost-effective  test  programs  and  support  of  UUT. 
The  document  describes  the  format  and  content  for  detailed  design  data  and  includes 
separate  E/E  and  F/I  test  requirements  sections  for  the  UUT.  This  document 
constitutes  the  formal  interface  between  the  contractor  responsible  for  detailed  design 
and  test  program  developer.  TRD  also  provides  detailed  configuration  identification 
for  UUT  design  and  test  requirements  data.  This  data  must  be  kept  current  with  the 
UUT  hardware  design  and  performance  changes  in  order  to  ensure  compatible  test 
programs.  It  must  be  a living  document  from  which  TPS  developer  can  obtain  source 
data  to  update  TPSs  for  required  UUT  configuration  levels.  The  initial  preparation  of 
TRD  should  also  involve  the  TPS  developer  to  ascertain  that  the  test  requirements  and 
strategy  are  properly  identified  and  noted  in  the  TRD.  This  version  of  the  TRD  should 
be  complete  when  the  configuration  of  the  first  preproduction  model  is  established  and 
should  contain  all  the  essential  elements  required  to  develop  effective  TPSs. 
Subsequent  revisions  to  the  TRD  should  be  processed  via  the  change  control  process 
identified  in  paragraph  2.5.  Initially,  the  TRD  for  end-to-end  and  F/I  test 
requirements  may  be  submitted  separately  for  accessing  the  ATE  data  bank  and 
commencement  of  TPS  development.  However,  there  is  a certain  degree  of  risk  and 
penalty  involved  in  terms  of  cost  and  schedule  impact  on  TPS  generation  when  the  ATE 
selection  process  reveals  different  ATEs  for  each  test  case.  The  obvious  objective  is 
to  select  one  common  ATE  which  must  satisfy  both  test  conditions.  Ideally,  the  ATE 
selection  process  should  be  based  on  the  combined  submittal  of  TRD  for  E/E  and  F/I 
testing  requirements  to  minimize  impact  on  TPS  development. 

2.3  ATE  selection.  The  process  of  selecting  off-line  ATE  has  been  added  and  is 
shown  as  part  of  TPS  development  activity.  This  task  involves  accessing  a family  of 
ATE  testers  and  corresponding  data  banks,  and  comparing  parametric  data  contained  in 
the  initial  TRD  document  to  arrive  at  a UUT/ATE  compatibility  matrix.  The  result  of 
this  compilation  provides  an  input  in  the  decision  process  in  selecting  the  most 
compatible  and  optimum  ATE  candidates  which  will  support  the  UUTs.  In  the  event 
the  family  of  ATEs  cannot  meet  UUT  test  requirements,  alternate  test  techniques  may 
be  requested  from  UUT  developer.  If  that  fails  to  resolve  the  problem,  then  the  UUT 
must  be  off  loaded  to  some  other  ATE  or  MTE.  A more  detailed  explanation  of  the 
process  in  selecting  ATE  can  be  found  in  the  Annex  which  follows  immediately. 

2.4  TPS  generation.  Test  program  generation  process  has  been  modified  in  order  to 
emphasize  the  main  tasks  that  are  performed  during  development  of  test  programs. 
There  are  three  major  areas  which  influence  test  program  design:  TRD,  ATE  user 
documentation  and  TPS  standards  and  design  policies  (e.g.,  AR-9B).  Since  the  TRD 
document  contains  precisely  defined  ATLAS  test  statements,  the  major  TPS  design 
effort  will  be  to  determine  overall  system  error  allocation  for  the  UUT  and  the  test 
set-up  including  the  selected  ATE.  Guidelines  with  respect  to  system  error  budgeting 
allocation,  i.e.,  test  accuracy  ratio  (TAR)  definition  should  be  provided  as  part  of  TPS 
design  policies.  The  output  of  test  analysis  results  in  detailed  step-by-step  test 
program  design  for  E/E  and  F/1.  In  addition,  the  analysis  also  determines  ATE/UUT 
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interface  compatibility  requirements  for  adapter  design  which  is  used  to  interface  the 
UUT  to  selected  ATE.  Common  adapter  design  should  be  utilized  to  interface  as  many 
UUTs  as  feasible  in  order  to  reduce  cost  and  proliferation  of  different  types  of 
adapters.  TPS  validation  and  verification  (V<5cV)  should  be  performed  on  first  article 
test  program  during  the  debugging  process.  V&V  should  be  accomplished  by  manual 
insertion  of  faults  in  the  UUT  to  demonstrate  the  capability  of  the  test  program  to 
successfully  detect  the  required  level  of  fault  isolation  on  the  UUT.  Preferably, 
automatic  fault  insertion  simulation  techniques  should  be  used  to  verify  fault  detection 
and  isolation  capability  of  the  test  programs.  Guidelines  with  respect  to  general 
acceptance  of  test  programs  should  be  generated  in  support  of  UUTs.  In  particular, 
the  issue  of  fault  insertion  and  detection  criteria  should  be  examined  and  established. 

2.5  Configuration  control  of  ATE,  TPS,  TRD  and  UUT  related  items.  NAVMATINST 


4 130.1  A identifies  requirements  for  configuration  management  in  terms  of 
identification,  control,  reporting  and  implementation  of  configuration  items  (CIs)  from 
validation  through  development  and  operational  phases.  In  particular,  there  is  a need 
for  a change  control  board  (CCB)  during  the  full  scale  development  phase.  The 
functions  of  CCB  are  to  review,  evaluate,  audit  and  approve/disapprove  Class  1 ECP 
changes.  The  formal  CCB  process  continues  during  other  acquisition  phases  except 
that  there  is  no  need  for  configuration  management  during  the  conceptual  phase.  In 
functional  terms.  Figure  1-2  identifies  CIs  which  should  be  under  configuration  control 
for  ATE  applications  and  how  it  may  affect  TPS/ATE/UUT  and  TRD  compatibility. 
Changes  in  any  one  of  these  areas  can  affect  the  maintenance  capability  for  the  UUT. 
Therefore,  from  the  configuration  point  of  view  whenever  changes  are  made  to  the 
ATE,  UUT  and  TRD,  it  is  essential  that  the  TPS  developer  be  notified  of  configuration 
level  changes  affecting  the  TPS.  Compatibility  can  only  be  established  by  strict 
adherence  to  configuration  control  procedures  which  must  be  established  for  ATE 
applications. 


3.0  SHORT  TERM  RECOMMENDATIONS 

o Finalize  TRD  and  obtain  concurrence  from  three  SYSCOMs  prior  to  end  of 
FY  77.  Ensure  that  this  document  contains  sufficient  and  accurate 
information  from  which  effective  end-to-end  and  F/I  tests  can  be 
generated  in  minimum  of  time  and  cost  by  the  TPS  developer. 

0 Identify  configuration  control  items  for  the  entire  ATE,  TPS,  UUT  and  TRD 
acquisition  process  and  show  the  relationship  and  impact  of  changes  to 
these  end  items.  If  feasible,  determine  standardized  reporting  method  for 
ATE  related  configuration  items. 

0 Establish  a methodology  for  the  number  of  faults  to  be  inserted  and 
sampled  during  TPS  verification  and  acceptance. 

o Determine  responsibilities  for  work  breakdown  structures,  e.g.,  TRA,  TRD, 
TPS  development,  etc.,  and  establish  policies. 


4.0  LONG  TERM  RECOMMENDATIONS 

o Establish  policies  and  procedures  for  the  management  of  TRDs. 

o Develop  tools  for  the  selection  of  ATE  from  a family  of  testers.  Presently, 

there  is  no  standardized  approach  to  be  used  in  the  selection  process. 

o Generate  design  guide  for  TPS  development  and  maintenance  (similar  to 
AR-9B)  which  will  be  applicable  to  three  SYSCOMs.  Also,  develop  design 
for  testability  document  (similar  to  AR-lOA). 

o Develop  a tool  which  will  provide  off-line  ATPG  data  that  is  ATE 
independent. 

o Establish  P&P  for  BIT  validation  after  qualification  testing.  As  an 

example,  BIT  was  not  validated  for  the  AWS  NAVSEC  Program.  Also,  BIT 
design  guide  NAVMATINST  3960.9  does  not  provide  standardized  strategy 
and  test  criteria  for  validation. 

o Generate  configuration  control  guidelines  for  all  ATE  related  items  based 
on  the  functional  flow  of  acquisition  activities  commencing  with  the 
product  baseline.  Include  the  guidelines  in  the  next  update  of  Acquisition 
Planning  Guide  as  an  appendix. 

5.0  REFERENCES 

(a)  Configuration  Management,  Department  of  the  Navy,  NAVMATINST 

4130.1A,  1 July  1974. 


(b)  Acquisition  Planning  Guide  for  Automatic  Test,  Monitoring  and  Diagnostic 


Structure,  ManTech  Systems  Report  prepared  for  MAT  04T, 
28  February  1977. 

(d)  Policies  and  Procedures  Analysis  - Validation  Phase  Work  Breakdown 


Structure,  ManTech  Systems  Report  prepared  for  MAT  04T,  14  March  1977. 

(e)  VAST  TPS  Cost  Investigation,  ManTech  Systems  Report  for  MAT  04T, 
30  December  1976. 

(f)  Report  on  Navy  Issues  Concerning  Automatic  Test,  Monitoring  and 


Diagnostic  Systems  and  Equipment,  Report  prepared  by  ATE  Ad  Hoc 


Working  Group  for  the  Assistant  Secretary  of  the  Navy  (Research  and 
Development),  13  February  1976. 

(g)  MlL-STD-881,  Work  Breakdown  Structures  for  Defense  Material  Items. 


(h)  Built-in-Test  (BIT)  Design  Guide,  NAVMATINST  3960.9,  9 September  1976. 
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1.0  ATE  SELECTION  PROCESS 


Figure  I-Al  describes  in  detail  the  process  of  determining  a workable  test 
scenario  from  alternative  options.  This  scenario  would  include: 

o Test  equipment  selection 

o Support  level  selection 

o Cost  prediction. 

There  are  four  main  phases  to  this  process: 

o Data  assembly 

o UUT/ATE  compatibility  analysis 

o Test  scenario  evaluation/cost  prediction 

o Report  and  recommendation. 

1.1  Data  assembly.  The  data  required  for  selection  of  candidate  ATEs  are  the  data 

establishing  the  need  to  test  a particular  UUT,  the  parametric  signal  data  and 
accuracies  required  at  the  UUT  I/O  interface,  and  the  corresponding  interface 
capability  of  candidate  ATEs.  The  sources  for  such  data  include; 

o Level  of  Repair  Analysis  (LORA)  for  UUTs 

o UUT  specifications/schematics 

o UUT  TRDs,  if  existent 

o ATE  interface  specifications. 

Other  data  to  be  assembled  at  this  time  should  include  ATE  cost  and  space 
requirements,  UUT  cost  and  MTBF  predictions  and  facility  locations.  These  data  are 
used  later  in  evaluation  of  logistics  cost  and  effectiveness. 

1.2  UUT/ATE  compatibility  analysis.  This  is  an  interactive  process  in  which  the 

j interface  requirements  of  each  candidate  UUT  (as  determined  by  LORA)  are  compared 

against  the  interface  capabilities  of  each  candidate  ATE.  In  general,  there  are  five 
alternatives  which  may  result  from  each  UUT/ATE  comparison: 

o The  ATE  is  not  compatible  with  the  UUT  and  cannot  be  made  so 
economically. 
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Figure  I-A1.  ATE  Selection  WBS  — Policies  and  Procedures  Item 


o Design-for-testability  modifications  can  be  made  to  the  (JUT  to  produce 
UUT/ATE  compatibility. 

o The  ATE  is  compatible  with  the  UUT  with  passive  interface  circuits. 

o The  ATE  can  be  made  compatible  to  the  UUT  with  modification. 

0 The  ATE  can  be  made  compatible  to  the  UUT  via  a complex  (e.g.,  active) 
interconnecting  devices  (IDs) 

In  the  cases  of  the  last  two  alternatives,  the  cost  elements  necessary  to  modify 
the  ATE  or  produce  the  complex  ID  must  be  estimated  and  carried  into  the  next  phase 
as  a part  of  the  support  equipment  costs. 

Identification  of  an  ATE/UUT  compatibility  match  is  completed  by  estimating 
UUT  test  time  and  by  indicating  the  UUT/ATE  compatibility,  in  matrix  form. 

1.2.1  UUT/ATE  matrix.  This  document  is  a three-dimensional  listing  of  UUT  versus 
ATE  support  levels  (O,  I,  D)  possible.  From  the  permutations  of  this  list,  possible 
test/support  scenarios  may  be  derived  for  evaluation  in  the  next  phase  of  the  ATE 
selection  process. 

1.2.2  Compatibility  analysis  outputs.  There  are  four  outputs  to  be  derived  from  the 
compatibility  analysis  phase: 

o The  UUT/ATE  compatibility  matrix 

0 The  identification  and  estimated  cost  of  required  ATE  mods  or  adapters. 

0 The  list  of  discard  UUTs,  i.e.,  those  not  considered  cost-effective 

candidates  for  test. 

o The  list  of  UUTs  which  must  be  tested,  but  whose  testing  is  not  cost- 
effective  on  ATE;  therefore,  candidates  for  PGSE  or  MTE  support. 

1.3  Test  scenario  analysis/cost  prediction.  The  purpose  of  this  phase  is  to 
systematically  determine  the  optimal  (i.e.,  most  cost-effective)  test  scenario  from  the 
options  developed  under  the  compatibility  analysis  phase.  This  is  done  by  judicious 
selection  of  a number  of  trial  support  scenarios  from  the  compatibility  matrix  and 
application  of  cost  estimate  analysis  to  each  selected  trial  scenario.  One  method  for 
such  analysis  is  the  test  equipment  effectiveness  model  (TEEM).  This  program, 
available  through  ESA,  utilizes  UUT  and  ATE  data,  shipping  and  labor  costs,  and 
selected  support  scenario  information  to  arrive  at  predictions  of  support  costs,  prime 
equipment  availability,  ATE  availability  and  manpower  and  spares  requirements.  The 
predictions  derived  for  various  selected  scenarios  may  be  used  as  a measure  of 
comparison  for  planning. 
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1.4  Recx)rt  and  recommendation.  There  are  three  primary  outputs  to  this  phase; 


o Final  Support  Recommendation  (Optimum  Scenario) 

o Cost  and  Availability  Predictions 

o Recommendations  for  Prime  Equipment. 

1.4.1  Final  support  recommendation.  This  output  consists  of  the  description  of  the 

recommended  support  scenario,  including; 

o Test  equipment  (ATE,  PGSE,  MTE)  selected 

o Sites  selected  for  test  equipment  use 

o UUTs  assigned  to  specific  test  equipments 

o Spares  recommendations. 

1.4.2  Cost  and  availability  predictions.  This  output  consists  of  the  estimates  of 
acquisition  and  operating  costs  for  prime  equipment  and  test  equipment  support.  It 
may  be  considered  as  the  non-recurring  and  recurring  costs  of  support.  Included  are 
manpower,  transportation  and  spares  cost  estimates. 

1.4.3  Recommendations  for  prime  equipment.  This  output  consists  of  any 
recommended  prime  equipment  changes  considered  necessary  to  make  the  support 
scenario  workable/more  cost-effective.  Examples  of  such  recommendations  might 
include; 

o Inclusion/augmentation  of  BIT 

o Design-for-testability  changes 

o Reliability  improvements. 

To  the  extent  that  the  inclusion  of  such  changes  to  the  prime  equipment  can  be 
incorporated  with  minimal  impact,  the  overall  life  cycle  cost  (LCC)  of  the  weapon 
system  may  be  reduced.  For  this  reason,  the  ATE  selection  process  should  be  integral 
to,  and  supportive  of,  the  prime  equipment  design  process. 


ATTACHMENT  1 


SUMMARY  OF  THE  TEST  REQUIREMENTS  DOCUMENT 


The  test  requirements  document  (TRD)  establishes  a definitive  and  precise  set  of 
requirements  for  UUT  source  documentation  and  procedures  required  in  support  of 
maintenance  systems,  subsystems,  assemblies,  and  subassemblies.  The  TRD  may  be 
used  to  perform  any  or  all  of  the  following: 


a.  Provide  a standard  data  package  which  can  be  used  to  analyze  a UUT  in 
sufficient  depth  to  facilitate  its  release  for  pre-production  and/or 
production  with  a high  level  of  confidence  that  provisions  are  provided  in 
the  design  to  align,  fault  isolate,  and  verify  equipment  operation. 

b.  Provide  a standard  data  package  to  the  developer  of  a test  program  set 
(TPS)  or  a test  procedure  so  that  programs  or  procedures  to  align,  fault 
isolate,  and  verify  equipment  operation  may  be  developed. 

c.  Develop  test  procedures  to  align,  fault  isolate,  and  verify  equipment 
ojseration. 

d.  Determine  the  test  equipment,  tools,  and  test  fixtures  required  to  support  a 
UUT  at  any  level  of  maintenance. 

e.  Provide  a vehicle  in  which  data  related  to  readiness  and  maintenance 
requirements  be  made  readily  available. 

This  document  consists  of  the  primary  specifications  and  four  appendices.  The 
purpose  of  each  appendix  is  as  summarized  below: 

Appendix  I - This  appendix  is  titled:  "Developmental  Documentation  and  Non- 
procedural Test  Requirements." 

This  appendix  provides  a standard  data  package  which  can  be  used  to  analyze  a 
UUT  in  sufficient  depth  to  facilitate  its  release  for  pre-production  and/or  production 
with  a high  level  of  confidence  that  provisions  are  provided  in  the  design  to  align,  fault 
isolate  and  verify  equipment  operation.  As  a standard  data  package  it  can  be  used  by 
the  developer  of  a test  program  set  or  a test  procedure  so  that  programs  or  procedures 
to  align,  fault  isolate,  and  verify  equipment  operation  may  be  developed.  It  is  a 
vehicle  for  determining  the  test  equipment  required  to  support  a UUT  at  any  level  of 
maintenance. 


Appendix  II  - This  appendix,  "Summary  Description  of  ATLAS,"  provides  an  overview  in 
the  utilization  of  ATLAS  as  a standard  abbreviated  English  language  in  the  preparation 
and  documentation  of  non-procedural  or  procedural  test  requirements. 


Appendix  III  - This  appendix,  "Application  of  UUT  Failure  Rate  Data,"  provides,  via 
standardized  forms,  the  capability  of  recording  reliability  failure  rate  data.  This  data 
can  be  utilized  to  effect  fault  isolation/diagnostic  test  design  and  also  provides  a 
^ vehicle  to  validate  the  attainment  of  fault  isolation  Ambiguity  Group  Requirements 

specified  by  the  procuring  agency. 


Appendix  IV  - This  appendix  is  titled:  "Production  Documentation  and  Procedural  Test 
Requirements."  This  appendix  is  similar  to  Appendix  1 but  it  provides  in  addition  to 
documentation,  detailed  procedures  to  align,  fault  isolate,  and  verify  equipment 
operation.  In  addition,  it  provides  a vehicle  for  determining  the  test  equipment,  tools, 
and  test  fixtures  required  to  support  a UUT  at  any  level  of  maintenance. 

Documentation  and  non-procedural  test  requirements  may  be  procured  as  per 
Appendix  I for  equipments  where  this  information  is  required,  but  configuration 
baseline  or  system  design  data  is  not  of  a mature  nature.  Data  also  may  be  procured 
as  per  Appendix  I where  the  procuring  agency  intends  to  utilize  the  documentation  and 
test  requirements  information  provided  as  first  generation  inputs  and  later  refine  them 
via  other  contractual  vehicles. 

Documentation  and  procedural  test  requirements  may  be  procured  as  per 
Appendix  IV  for  those  equipments  where  the  procuring  agency  believes  that  a 
minimum  amount  of  risk  is  associated  with  contracting  directly  for  procedures  based 
on  equipment  maturity  or  other  factors.  Where  equipment  is  of  an  immature  nature 
but  the  procuring  agency  is  in  need  of  procedural  test  requirements,  data  may  be 
procured  as  per  Appendix  IV  but  this  data  should  be  monitored  and  controlled  by  a 
suitable  configuration  control  plan  of  action. 

Throughout  this  specification  the  utilization  of  the  ATLAS  language  for 
describing  procedural  and  non-procedural  test  requirements  is  specified  regardless  of 
whether  ATE  or  semi-automatic  test  equipment  is  contemplated  to  be  utilized. 

ATLAS  is  an  accepted  industry  standard  and  is  dedicated  to  defining  the  test 
requirements  of  a unit  under  test  with  no  reference  to,  nor  dependence  upon  the  test 
equipment  which  may  be  used.  For  those  instances  where  ATLAS  cannot  adequately 
describe  test  requirements  in  an  unambiguous  fashion,  this  specification  defines 
certain  minimum  data  requirements  which  must  be  supplied  as  part  of  the  TRD  to 
rectify  these  anomalies. 
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